Above a certain size, SEO stops being about individual pages and becomes about systems. You cannot review a hundred thousand URLs by hand, so the work shifts to templates, patterns and sampling — and the problems change shape accordingly.
What changes at scale
- Crawl capacity becomes finite in practice. Not everything gets crawled often, so what you spend it on matters.
- Template faults multiply. One bad canonical rule affects fifty thousand pages at once.
- Indexation becomes partial by default. A large share of URLs may simply never be indexed, and some of that is correct.
- Internal linking does the distributing. With too many pages to link from navigation, structure decides what gets found.
- Manual review stops working. You sample and you measure; you do not inspect.
The dominant problem is usually too many URLs
Large sites rarely suffer from having too few pages. They suffer from generating far more URLs than they have content — parameters, filters, sorts, session identifiers, near-duplicate variants.
Every one of those consumes crawl capacity that could have gone to pages you want indexed. Reducing the URL surface is generally the highest-return work available. See index bloat and duplicate URLs.
Auditing by template, not by page
- Group URLs by template and sample each group.
- Check indexation rates per group. A template with a very low rate is the finding.
- Check crawl frequency per group against commercial value — the mismatch tells you where capacity is going.
- Treat findings as template faults, because that is what they almost always are.
A page-level list of a hundred thousand issues is unusable. A list of twelve templates with the number of URLs each affects is actionable.
Log files earn their place here
On a small site, log analysis is a luxury. On a large one it is the only way to see what crawlers actually did rather than what you assume — which sections get attention, which are ignored, and how much capacity goes to URLs that should not exist.
See log file analysis.
Depth and linking
Pages far from the homepage are crawled rarely enough that indexation becomes unreliable. On a large site this is structural rather than incidental — pagination depth, category nesting and weak cross-linking all push pages further away.
See internal linking and crawl depth.
Frequently asked questions
How large is large?
When you can no longer review URLs individually and indexation is visibly partial. In practice, tens of thousands of URLs upward.
Should every page be indexed?
No. On a large site, deliberately keeping low-value URLs out of the index concentrates attention on the ones that matter.
Is crawl budget real?
It is not a published allowance, and crawl capacity is finite and observably unevenly distributed. See crawl budget.
Where do I start?
Count your URLs, then count how many you actually want indexed. The gap between those numbers is usually the work.
Consulting CTA
For a template-level audit of a large site, book an SEO consultation.