Server logs record every request your server received, including every request from a search engine crawler. That makes them the only source that shows what was actually crawled, rather than what could be crawled or what a tool inferred.
For most sites this is unnecessary detail. For large sites with crawl problems it is the difference between diagnosis and speculation.
What logs show that nothing else does
- Which URLs were actually requested, and how often.
- What was returned, including errors that tools sampling a crawl would miss.
- How crawl activity is distributed across URL patterns, which is the key insight.
- Response times as experienced by the crawler, which affects crawl rate directly.
- Which crawler, distinguishing Google from other bots and from fake user agents.
Search Console crawl stats give a summary of some of this. What they do not give is the per-pattern breakdown, which is where the actionable finding usually lives.
The question logs answer best
Where is crawl capacity going? On a large site the answer is frequently uncomfortable: a large share consumed by filter URLs, parameters, redirects and pages nobody should be crawling, while genuine products wait.
- Segment requests by URL pattern and compare against your real page inventory.
- Compare crawl share against revenue share by section.
- Identify patterns being crawled heavily that should not be crawled at all.
- Find pages that have not been crawled at all in the period.
That last one is valuable and hard to get elsewhere. A page never requested cannot be indexed, and no crawl tool will tell you it was skipped.
When it is worth doing
- Large catalogues where crawl budget is a genuine constraint.
- Sites where new pages take a long time to be indexed.
- After a migration, to confirm redirects are being crawled and old URLs processed.
- Where Search Console reports large numbers of discovered-but-not-crawled URLs.
- Where crawl activity has fallen and the cause is not obvious.
When it is not
Small sites, sites with no indexation problems, and sites where the obvious causes have not yet been checked. Log analysis is a precision instrument and it is not the first thing to reach for. Check robots.txt, directives and internal linking first.
Practical obstacles
- Access. Shared hosting frequently does not provide raw logs, and CDN logs may be separate or require a higher plan.
- Retention. Logs are often rotated within days, so you may need to arrange retention before you can analyse a period.
- CDN interception. Where a CDN serves cached responses, origin logs miss those requests entirely, which distorts the picture.
- Verification. Crawler user agents are trivially spoofed, so requests should be verified by reverse DNS before being counted.
Frequently asked questions
Do I need log analysis?
Most sites do not. It becomes valuable on large sites with crawl constraints, or where indexation problems persist after the obvious causes have been ruled out.
Is Search Console enough?
For most purposes, yes. Its crawl stats give totals and trends. Logs give the per-pattern detail that turns an observation into an action.
How much log data do I need?
Enough to cover a representative period, typically two to four weeks. Longer for sites crawled infrequently, since a short window may not include the pages you are asking about.
What if my host will not provide logs?
Check the CDN, which frequently offers request logs. Failing that, Search Console crawl stats plus a thorough crawl will answer most questions less precisely.
Consulting CTA
If your large site has persistent indexation problems and the obvious causes are ruled out, book an SEO consultation.