Crawl Budget and Discovery on a Corporate Estate

On a corporate estate the indexing problem is never that a page cannot be found. It is that eleven thousand pages can be found, four hundred of them matter, and nobody in the building can produce a list of which four hundred. Everything that follows from that — wasted crawling, stale documents outranking current ones, new pages appearing weeks late — is a consequence of the missing list, not of the search engine's behaviour.

Zurich has an unusual density of this shape: regional or global headquarters running one domain that carries country folders, product documentation in several versions, an investor section, a press archive going back a decade and a careers area that a different vendor operates. Each part was built by a different team, at a different time, under a different assumption about what should be public.

This piece walks the discovery chain on an estate of that size and describes where it breaks. The mechanics refer to the indexing area of the Semalt panel; the sequencing applies regardless of which tool submits the URLs.

At a certain size the problem inverts

On a small site the task is to be found. On a large one the task is to be found selectively. Crawling capacity spent on a decade of press releases is capacity not spent on the product pages published last week, and unlike a small site, a large one can absorb that waste indefinitely without anything looking obviously wrong.

• Discovery The address is known — via a link, a sitemap or a submission. Cheap and rarely the bottleneck here.
• Fetching The crawler retrieved it. On large estates this is where capacity gets consumed.
• Selection The page entered the index. An editorial judgement, and the step that most corporate pages fail.
• Refresh The page is revisited after changes. The forgotten fourth step, and the reason stale prices persist for months.

The per-URL log makes this visible without touching server logs: the bot visit with its timestamp, the status, and the error detail if there was one. For most corporate teams that record is the first time anyone can answer, with evidence, the question of whether a given page was ever actually fetched.

Four generators of addresses nobody ordered

Large estates rarely grow by publishing. They grow by generating. Four mechanisms account for the overwhelming majority of the excess, and all four are invisible in the content management system, because from its point of view nothing was created.

The second pair is harder to resolve than the first, because both mechanisms serve real purposes. Old documentation matters to customers still running that release. Country pages exist because local rules and contact details differ. The answer is not deletion but designating which version is the canonical destination and making the others clearly subordinate.

The inventory nobody has, and how to build it in a day

Every remedy in this article depends on a list of the addresses that are supposed to exist. Almost no large organisation has one, and building it is less work than it sounds because three partial lists already exist and only need reconciling.

Reconciling three lists sounds like an audit and is in practice a morning with a spreadsheet. The output is a table of path prefixes with a count for each of the three sources, and the rows where the counts diverge point straight at the generator responsible. In estates of ten thousand addresses it is common to find that two prefixes account for eighty percent of the excess.

Remove, connect, submit, then measure

The order matters more than the speed. Each step makes the next one cheaper, and doing them in the wrong sequence means repeating work.

• Remove Decide what should not be reachable at all — sort orders, internal search results, print views, retired campaign pages.
• Connect Ensure every page that should exist is reachable from at least two others, not only from a sitemap.
• Submit Only what is new or genuinely changed, in batches, within the daily allowance.
• Measure Take a sample of thirty addresses across sections and read their logs before drawing conclusions.

The connect step is the one corporate estates skip, and it is the one that matters most. A product page linked only from a sitemap and a mega-menu is structurally weaker than a page linked from three related articles. On a large estate this shows up as whole sections that are known but never selected, and no amount of submission changes it.

One number is worth checking before any of this begins, because it changes the plan. Nested sitemaps are read three levels deep and a single job handles up to a thousand maps. That is generous for almost any estate, but corporate sites that generate a map per country and per product line occasionally nest a fourth level — and everything below that level is simply never read. The symptom is a whole territory absent from the report with no error attached to it, which reads like a mystery until somebody counts the levels.

Counting is the underrated skill in this whole exercise. Where the three lists disagree by more than ten percent, one of the three systems holds a picture of your website that is wrong, and an hour of sampling will say which. Teams that do this once tend to keep doing it, because it converts an argument about impressions into a question with an answer. The campaign side, once the counts agree, is where the managed tier earns its place — target pages on an estate this size are worth choosing individually rather than by rule.

Country folders, and the duplication that is not duplication

Eleven country versions of a product page are a legitimate structure and a reliable source of confusion. The pages are near-identical by design; the differences that matter — a local phone number, a compliance note, a price in a different currency — are exactly the parts a similarity comparison discounts.

The third row is the uncomfortable one. Not every market needs its own page. A country version that differs only in a footer belongs merged into a regional page that names all the markets it serves. That decision is organisational rather than technical, which is why it takes longer than the fix itself, and why the technical work should not start until it is made.

Documentation, archives and the freshness question

Versioned documentation raises a question small sites never face: which release should be the one people find? Publishing every version equally means the search results for your product show a manual from three releases ago as often as the current one, and support tickets follow.

Archives raise the mirror-image question. A press section that has been running for a decade contains a great deal of text about the company, written when different things were true. Some of it is historically valuable and some is simply outdated, and search engines have no way to tell which is which — they see recent, well-linked, on-topic documents. The editorial answer is to date everything visibly and to keep archive items out of the internal link paths that current pages depend on. Where that housekeeping belongs in the wider content plan is covered under content work.

The workable pattern is a stable address for the current version, with versioned addresses subordinate to it and clearly labelled. Customers on older releases still reach what they need; everyone else lands on what is current. The same logic applies to a press archive: the archive stays reachable, but nothing in it competes with a current product page for the same query.

What has to keep running afterwards

The cleanup is a project; keeping the estate clean is a routine. On a corporate site the routine has to survive staff changes, agency changes and a content management system that generates addresses on its own, which means it must be small enough that nobody drops it.

Handing the submission step to whoever deploys, rather than to whoever reports, is the single change with the largest effect. When it depends on a request to an external party it happens in batches every few weeks, and the fast publication the whole exercise was meant to achieve quietly disappears. Where campaign work runs alongside, the same principle applies under automated campaign management.

A realistic schedule and what improves first

On an estate of five to fifteen thousand addresses, expect six to ten weeks between the first cleanup and a clearly different picture. The first improvement is not traffic. It is that the reports become readable: the count of known-but-not-selected addresses falls, and the remaining rows are ones somebody can act on.

The second improvement is publication speed. Measure it deliberately — note the publication date and first impression for the next ten pages — and you will have a figure specific to your own estate. Well-maintained corporate sites tend to land at two to four days; neglected ones at two to four weeks. That difference is almost entirely the work described in section four, and it is the number that makes the case internally, because a delay is a scheduling problem and scheduling problems get decided faster than quality problems.

Limits worth stating plainly

There is also a side effect that is worth anticipating politically. A clean inventory makes gaps visible that were previously hidden in the noise: a country folder untouched for two years, a documentation set that stops at an old release, an entire section whose owner left the company. That is the real return on the exercise, and it lands as a list of uncomfortable facts rather than as a technical report. Deciding in advance who receives that list, and in what form, avoids a great deal of friction later.

If you want the current state of your own estate, connect the domain in the Semalt panel and submit nothing at all for the first week — read the report as it stands. That untouched picture is the most honest inventory available in an hour, and it is a better starting point than any plan written before it. Further pieces sit in the blog, and the surrounding market view is covered under generated market research.