Crawl budget is the amount of crawling a search engine will do on your site in a given period. On small sites it is not a constraint. On large catalogues it becomes the limit on how much of your store stays indexed and how quickly changes are noticed.
The practical question is not how to increase the budget, which is largely outside your control, but how to stop spending it on URLs that will never earn anything.
Where index bloat describes the surplus that ends up indexed, crawl budget describes the capacity consumed producing it. The two usually appear together and share most of their remedies.
When it starts to matter
- Catalogues beyond roughly ten thousand crawlable URLs, which many mid-sized stores exceed through filters alone.
- Stores where filter and parameter URLs outnumber genuine pages.
- Sites with slow server responses, since crawling rate adapts to what the server can sustain.
- Frequently changing catalogues where recrawl speed determines how current the index is.
- Any store where new products take weeks rather than days to appear.
A store with two hundred products and no faceted navigation will not have a crawl budget problem. A store with two hundred products and five filters might.
Where crawling gets wasted
- Filter combinations, which multiply rather than add and are the largest single waste on most stores.
- Redirect chains, where each hop is a separate request. Chains after a migration are a common cause.
- Soft 404s, where missing products return a page rather than a status code, so crawlers keep requesting them.
- Slow templates, since crawl rate adjusts to server response time.
- Stale sitemaps listing URLs that no longer exist.
- Infinite spaces such as calendars or crawlable internal search.
Diagnosing it properly
Crawl budget analysis depends on server logs. Third-party crawl tools show what is crawlable; only logs show what was actually crawled.
- Sample server logs and segment requests by URL pattern.
- Compare crawl requests against revenue by page type. The mismatch is usually stark.
- Measure how much crawling goes to parameter URLs versus products and categories.
- Check crawl frequency on priority categories specifically.
- Watch average response time, since it caps crawl rate directly.
Search Console crawl stats give a usable approximation where logs are unavailable, though without the per-pattern detail that makes the diagnosis actionable.
Recovering wasted capacity
- Prevent crawling of filter and sort URLs at source rather than after they are indexed.
- Flatten redirect chains so every redirect resolves in one hop.
- Return correct status codes for removed products instead of soft 404s.
- Keep sitemaps accurate and current, since they guide discovery.
- Improve response time on category templates, which are usually the slowest.
- Remove internal links to URLs you do not want crawled, because links are followed regardless of directives.
Structural relief matters too. A well-organised hierarchy reduces the crawling needed to reach the catalogue; see online store site architecture.
Frequently asked questions
Can I increase my crawl budget?
Only indirectly. Faster responses and a healthier site tend to earn more crawling, and more valuable content earns more attention. There is no setting. The reliable gain is spending less of it on worthless URLs.
Does crawl budget affect small sites?
Rarely. A few hundred pages will be crawled comfortably. It becomes a constraint when crawlable URLs run into the tens of thousands, which faceted navigation can cause on a modest catalogue.
Do I need server logs?
For a proper diagnosis, yes. Crawl tools show what could be crawled; logs show what was. Search Console crawl stats are a reasonable substitute for direction, but lack the per-pattern breakdown.
Will blocking filters hurt users?
No, if done correctly. Filters continue to work for shoppers. You are preventing crawlers from treating each combination as a separate page, which is a different thing from disabling the feature.
Consulting CTA
If new products are slow to index, or your crawl activity is dominated by filter URLs, book an SEO consultation for a crawl efficiency review of your catalogue.