Washington organizations publish more documents than almost anyone, and retire almost none of them. The result is an archive of real intellectual value that a crawler has largely stopped visiting, and the reason has nothing to do with quality.
Most website work assumes publishing is the finish line: put the report up, link it from a landing page, and the rest follows. For an association with thirty years of proceedings or a think tank with four thousand PDFs it does not follow, and the failure is quiet enough that nobody notices for years.
The archive nobody has ever pruned
This is the association capital of the world, and associations publish. So do the think tanks along Massachusetts Avenue, the advocacy groups, the standards bodies and the professional societies. Between them: testimony, comment letters, annual reports, conference proceedings, member directories, issue briefs and decades of press releases — nearly all still online, nearly all still linked from somewhere, and almost none ever reviewed for whether it should be.
Two habits make this particular. The document is usually a PDF behind a thin landing page repeating its title over a download button. And nothing is ever removed, because deleting an organization's own record feels like destruction rather than maintenance. Both instincts are defensible. Together they build a site where the count of addresses grows every quarter and the count worth requesting does not.
- Publication is treated as permanent. A 2009 press release announcing a staff hire is still a live page, still linked, still consuming attention that belongs to this year's work.
- The landing page duplicates the document. Two addresses for one piece of thinking, and the one a crawler reads most easily is often the emptier of the two.
- Nobody owns retirement. Publishing has a process and a calendar. Removal has neither, so it never happens.
- The valuable part is genuinely valuable. This is not a junk archive. It is a serious one that has been left to compete with itself.
A page that exists is not a page anybody has looked for
Three separate events have to happen before a document can appear in results, and treating them as one is where the confusion starts. A crawler has to learn the address exists. It has to fetch it. Then another process decides whether what it fetched is worth keeping. Failure at any of the three looks identical from outside: the page is not there.
Most archive problems live at the first stage. A comment letter from 2014 is reachable in that a path of links leads to it, but the path runs through page eleven of a paginated index nothing has fetched in a year. It exists. It has effectively never been offered.
| Stage | What has to happen | Typical failure in an archive | Visible symptom |
|---|---|---|---|
| Discovery | The address becomes known | Buried in deep pagination, listed nowhere else | No record of any visit |
| Fetch | The document is retrieved | Ration spent on near-identical pages first | Visited once, years ago |
| Retention | It is judged worth keeping | A landing page that only restates a title | Fetched, then quietly dropped |
| Refresh | It is revisited as it changes | Nothing signals that anything changed | An outdated version persists |
The distinction matters because the remedies are different. A discovery failure is fixed by telling somebody the address exists. A retention failure is not fixed that way at all, and repeating the request only produces a faster refusal.
What actually consumes the attention your site receives
A crawler allocates finite work to any one domain, roughly in proportion to how useful previous visits proved and how quickly the server answers. That allowance is published nowhere and cannot be bought. It can only be spent well or badly, and large archives are unusually good at spending it badly.
Addresses that multiply on their own
The publication library with filters for year, topic, author and format generates a distinct address for every combination.
- Sort orders and view toggles
- Session and tracking parameters
- Endless pagination with no terminus
Pages that restate each other
Landing pages whose entire content is a title, a date, a two-line abstract and a download link.
- Both a summary page and the PDF
- Press releases in three formats
- Event pages for events long past
Documents with substance on the page
A report whose findings, methodology and conclusions are readable as HTML, with the PDF offered alongside.
- Text a reader can quote from
- A visible, unambiguous date
- Named authors and sources
Structure that says what matters
A hierarchy where the depth of a document reflects how current and how important it is.
- Current work close to the root
- Archive clearly marked as archive
- Retired material removed, not hidden
Server speed belongs here too, and document-heavy sites lose on it without knowing. A search interface taking four seconds to assemble a listing teaches a crawler to come back less often. The archive did not slow down because it grew; it slowed because nobody revisited the query behind the listing after the ten-thousandth record went in.
The document is a PDF and the page in front of it is empty
This is the commonest structural problem in a Washington document library. The substance — forty pages of analysis with footnotes — lives in a PDF. The page linking to it holds a title, a date, a paragraph and a button. Two addresses for one piece of work, and the thinner one is presented as the destination.
PDFs are readable and can rank, so this is not a catastrophe. But a PDF is a poor landing place: it cannot be navigated, it links to none of your other work, and on a phone it is close to unusable. Meanwhile the page that could have carried the substance carries almost nothing — which is precisely what a retention process declines to keep.
- Put the substance on the page. Summary, key findings, methodology and conclusions as readable text, with the PDF offered as the formal citable version.
- Date everything visibly. An audience that cites sources for a living will not use a document it cannot date, and neither will anything reading the page.
- Do not offer the same work twice. Decide which address is the destination and make the other one point at it rather than compete with it.
- Name the authors. In a market of analysts and researchers, an unattributed report is a report nobody quotes.
The sitemap is an instrument of discovery, not an inventory
A sitemap is usually treated as a dump of every address a system can enumerate. It is more useful read as a claim: these are the documents we consider worth somebody's time. Whatever you could not defend in a meeting belongs outside it, and leaving it out is a decision rather than an oversight.
Structure follows from that. A large archive should not be one enormous file but an index pointing at sections — current publications, testimony, events, member resources, the archive itself — each with its own file, so a fault in one part reads as a fault in one part. Recursive parsing follows nested files three levels deep, which is enough for the largest library in this city.
Sitemap jobs
For the case where the question is whether anything knows these addresses exist at all.
- Nested files are followed automatically. Recursive parsing to three levels, so an index of indexes resolves without manual flattening.
- Scale suits a real archive. Up to 1,000 sitemaps in a single job, which covers a document library of essentially any size a membership organization will build.
- Jobs queue rather than collide. Two run at once and up to twenty wait, so a full-archive pass and a weekly update do not have to be scheduled around each other.
Treat the first job on an untended archive as an audit rather than a submission. It reports how many addresses your own system believes exist, and that figure is often several times what anyone inside would have guessed. The gap is filters, pagination and duplicate formats, generated automatically over years by a content system nobody questioned.
A thousand a day, ten thousand in a batch
Alongside sitemaps sits direct submission, which asks for specific addresses to be looked at. The limits are firm and worth knowing before planning around them: 1,000 URLs per day per account, and up to 10,000 in a single batch. A batch larger than a day's allowance is not rejected — it works through the queue at the rate the daily budget permits.
Bulk URL submission
For addresses you have deliberately chosen, not everything a crawl of your own site returns.
- Requests go out over IndexNow. The protocol notifies participating crawlers, GoogleBot and BingBot among them, rather than waiting for a scheduled visit.
- Every address gets its own record. Bot visit with a timestamp, resulting status, and error detail where something failed.
- Counters run while the batch does. Submitted, found and failed update live, so a systematic problem is visible in the first minutes rather than the following week.
That distinction decides how the daily allowance should be spent. It goes to work that genuinely changed — new testimony, a rewritten report page, a merged set of duplicates, a redirect that now needs following. It does not go to a standing job that resubmits the same four thousand archive pages every month. Those pages have been seen. Sending them again tells nobody anything, and the ration you spent on them was the ration this week's publication needed.
Reading the status of a batch
The per-URL record is the most useful output here and the least read. Each address carries the visit that happened, when it happened, what status came back and what went wrong if anything did. Aggregated, that log answers a question no ranking report can: is this a discovery problem or a retention problem?
| What the log shows | Most likely reading | What to do about it |
|---|---|---|
| No visit recorded at all | The address was never discovered | Get it into a sitemap and link it from a page that is visited |
| Visited, then never returned to | The first visit did not justify a second | Put real content on the page or retire it |
| Repeated server errors | The archive query is timing out | Fix the listing before submitting anything else |
| Redirect chains several steps long | Years of reorganizations layered on each other | Point the first address at the final one directly |
| Failures clustered in one section | One subsystem, not a site-wide fault | Treat it as one bug, not four thousand |
A discovery problem
The log shows no visit, or one visit long ago. Nothing knows these addresses are worth returning to.
- Fixed by sitemaps and internal links
- Submission genuinely helps here
A retention problem
The log shows visits that led nowhere. The address was seen, judged and set aside.
- Fixed by the page, not the request
- Resubmission changes nothing
There is a jurisdictional wrinkle worth noting for the membership side of these sites. Chapters and resources are frequently organized by state, and in this market that means the District, Virginia and Maryland, which differ on licensing, tax treatment and what a member may be told about compliance. Those pages should be genuinely separate documents, not one page with three headings, and a discovery report that lumps them together will hide a problem affecting two thirds of the membership.
Running the numbers on a dormant archive
Take an association with twenty-eight years of output. Four thousand two hundred documents, each with a landing page and a PDF: 8,400 addresses. Filters on year, topic and format generate perhaps 1,900 more. Event pages for conferences since 1998 add 340. Press releases, at forty a year, contribute 1,100. The reachable total clears 11,700 — and the organization would have told you it had about four thousand pages.
At the daily allowance the whole thing is submitted inside twelve days, and one batch holds most of it. The limits were never the obstacle. The obstacle is that submitting all 11,700 requests attention for 1,900 filter combinations and 1,100 staff-appointment releases, at the expense of the four hundred documents anybody still reads.
The triage is a conversation, not a technical exercise. Ask the librarian, the research director and whoever answers the members' inquiry line which documents get requested. The answer is usually a few hundred: the standards, the annual survey, the testimony people still cite, the current guidance. Those get real pages with the substance on them. The next tier stays as it is, in a labeled archive that stays reachable and searchable without competing. Filter combinations leave the sitemap entirely, and pre-2015 press releases become one dated index page rather than 1,100.
- Current work sits near the root. Anything published this year or still authoritative should be two clicks from the front page, not eleven pages into a filtered listing.
- The archive is labeled as an archive. Reachable, searchable, dated, and honestly presented as historical rather than dressed up as current.
- Generated combinations leave the sitemap. Filters and sort orders are navigation. They are not documents and should never have been offered as such.
- Retirement gets an owner and a date. Whoever runs publication also runs removal, on a schedule, or the count grows forever.
Four hundred well-presented documents beat 11,700 that dilute each other, and this is not an editorial argument. It follows from a fixed ration and from what a crawler learns to expect of this domain. Anyone weighing whether to audit before cleaning up generally should: the audit is what makes the triage conversation concrete.
Questions that come up on the first archive pass
Our archive has real historical value. Are you telling us to delete it?
No. Keep it and mark it as what it is. The problem is not that old testimony exists; it is that it looks identical to this month's work, so nothing distinguishes the two. An archive section that is reachable, searchable and clearly dated preserves everything and stops it competing with current publications.
We submitted 3,000 archive URLs and nothing changed. Why?
Submission covers the first stage only, and the blockage was almost certainly later. Landing pages holding a title and a download link get requested and then dropped; asking again changes the speed of the outcome and nothing else. Check the per-URL log: if those addresses were visited and never revisited, the issue is what is on them.
Should the PDF or the landing page be the destination?
The page, for anything you want read and cited. Put the substance in HTML and keep the PDF as the formal version for people quoting it with page numbers. You get a destination that can be navigated, linked internally and read on a phone, while the citable artifact stays where researchers expect it.
What do we do with fifteen years of press releases?
Judge them by whether anything specific happened. A release announcing a substantive position or a major publication is a document and stays. A staff appointment from 2011 is a record, not a document: fold those into a dated index page. The history survives and a thousand addresses that each answer nothing disappear.
How often should we run a sitemap job once the cleanup is done?
On publication and after structural changes, not on a timer. A new report, a merged section, a batch of redirects — those are worth announcing. Re-announcing an unchanged archive every week consumes the daily allowance and tells nobody anything they did not already record.
Start by finding out what your own site thinks it contains. Connect the domain, run a sitemap job over the archive in whatever state it is in today, and read the per-URL log before editing a single page. Run a discovery pass over your document library; the same panel covers this whether you sit on the automated tier at $149 a month or the reviewed one at $500. The first finding is rarely about rankings. It is usually several thousand addresses nothing has ever asked for, generated by a publication system during a decade nobody was watching, quietly absorbing the attention that belongs to the work people still cite.