Everything else in SEO assumes your pages are crawled and indexed. When they are not, content quality, keywords and links are all irrelevant, because the page is not in the running at all.
This is where diagnosis should start whenever pages are missing from search.
Crawling and indexing are separate
They are routinely treated as one process and they are not, which is why diagnosis so often goes wrong.
- Crawling is a search engine fetching the page.
- Indexing is deciding to store it and make it eligible to appear.
A page can be crawled and deliberately not indexed. A page can be indexed without recent crawling. Blocking crawling does not remove a page from the index — it prevents the crawl that would discover the instruction to remove it, which is why robots.txt is the wrong tool for deindexing.
The usual causes
- Blocked in robots.txt, often inherited from staging.
- A noindex directive, sometimes added by a theme or plugin rather than deliberately.
- No internal links, leaving the page unreachable. See internal linking.
- Canonicalised elsewhere, so the page is treated as a duplicate.
- Judged not worth indexing, which is a content problem rather than a technical one.
- Simply not reached yet, which is common on large or new sites.
The last is worth ruling in before assuming a fault. A large site is crawled progressively, and steady growth in indexed pages means the process is working.
Guides in this section
- Website not indexed — when nothing is appearing at all.
- Google not crawling your website — diagnosing a stalled crawl.
- Submitted URL not indexed — what the Search Console status means.
- Google chose a different canonical — when your canonical is overruled.
- URL parameters — controlling parameter-generated URLs.
- Orphan pages — finding pages nothing links to.
- Duplicate URLs — one page reachable at several addresses.
- Sitemap not being read — diagnosing sitemap fetch and parse errors.
How to diagnose properly
Use URL Inspection in Search Console on a specific affected URL. It reports the actual reason — blocked, noindex, canonicalised, crawled and not indexed, or not discovered. Every one of those points at a different fix, and guessing between them wastes time.
Where the report says the page was crawled and not indexed, the cause is usually judgement about the content rather than a technical fault. See pages not indexed by Google.
For a full crawl and indexation review, see crawl and indexation audit or book an SEO consultation.