Crawling and indexing are separate steps, and confusing them sends people looking in the wrong place. Crawling is Google fetching your pages. Indexing is deciding to include them. If crawling is not happening, nothing downstream can.
This page covers the crawling problem specifically. If pages are being crawled but not included, see crawled, currently not indexed instead.
Confirm crawling has actually stopped
Search Console crawl stats show requests over time. Server logs, where available, are more precise because they show exactly what was requested and what was returned.
- A gradual decline usually points at site health or reduced perceived value.
- A sudden stop usually points at a specific change: robots.txt, server configuration or a firewall rule.
- Crawling concentrated on a few URLs while others are ignored is a prioritisation problem rather than a blockage.
Things that block crawling outright
robots.txt
A Disallow rule covering the paths in question. Read the file directly rather than relying on a plugin’s summary of it, and check for rules that apply to specific user agents.
Server errors
Repeated 5xx responses cause Google to reduce crawl rate to avoid making things worse, and sustained errors can stop it substantially. Check error rates before assuming an SEO cause.
Firewall and bot protection
Security services and CDN rules sometimes block or challenge crawlers, particularly after a configuration change. This is a common and easily missed cause, because the site works perfectly for humans.
Authentication
Password protection or IP restriction left on after a staging period. The site loads for you because you are logged in or on an allowed address.
Things that reduce crawling without blocking it
- Slow responses. Crawl rate adapts to what your server sustains, so a slow site is crawled less.
- Wasted capacity. Filter, parameter and redirect URLs consuming the budget that real pages need. See crawl budget.
- Low perceived value. Sites where most crawled content proves not worth indexing tend to be crawled less over time.
- Poor internal linking. Pages linked from nowhere are hard to reach and easy to deprioritise.
- Excessive depth. Pages many clicks from the homepage are crawled less frequently.
What to check, in order
- Read robots.txt in full.
- Use URL Inspection with a live test, which shows what Google actually receives right now.
- Check server error rates and average response time.
- Check CDN or firewall logs for blocked crawler requests.
- Compare crawl requests by URL pattern to see where capacity is going.
- Check internal links and click depth for the pages being missed.
What does not help
- Submitting the sitemap repeatedly. Sitemaps aid discovery, not crawl capacity.
- Requesting indexing for large numbers of URLs.
- Publishing more pages while existing ones are not being crawled.
Frequently asked questions
How often should Google crawl my site?
It varies enormously with size, update frequency and authority. What matters is whether your important pages are crawled often enough that changes are noticed, not an absolute rate.
Does crawl budget affect small sites?
Rarely. A few hundred pages will be crawled comfortably. It becomes a constraint at tens of thousands of URLs, which faceted navigation can produce from a modest catalogue.
Can I force Google to crawl?
You can request individual URLs, which prompts a fetch. You cannot increase overall crawl capacity directly. The reliable levers are faster responses, better linking and less wasted crawling.
Why is Google crawling my old URLs?
Google revisits known URLs for a long time, particularly ones that once had value. Ensure they return correct status codes rather than soft 404s, which keep them in rotation indefinitely.
Consulting CTA
If crawling has fallen and the obvious blocks are not present, book an SEO consultation for a crawl efficiency review.