Crawlability is whether a search engine can fetch a page. Indexability is whether it is allowed and chosen to include it. They are sequential, and a page can fail at either stage for entirely different reasons.
Almost every indexation problem becomes tractable once you establish which of the two is failing, because the remedies share nothing.
The two stages
Crawlability: can it be fetched?
- Is it linked from somewhere, or otherwise discoverable?
- Does robots.txt permit the request?
- Does the server return the page rather than an error?
- Is it behind authentication or a firewall rule?
- Is there crawl capacity available to reach it?
Indexability: will it be included?
- Does a noindex directive prevent it?
- Does a canonical point elsewhere?
- Is it a duplicate of something already indexed?
- Is there enough of substance to justify inclusion?
Note that a page can be perfectly crawlable and not indexable, and vice versa. The combination that confuses people most is crawlable, indexable, and still not indexed, which means it was assessed and passed over.
Which is failing
Search Console states this, and the exclusion reason maps directly onto the two stages.
- Crawl failures: Blocked by robots.txt, server error, not found, redirect error. See Google not crawling your website.
- Never reached: Discovered, currently not indexed. Crawlable in principle, not yet fetched.
- Index refusals: Excluded by noindex, alternate page with canonical, duplicate.
- Assessed and declined: Crawled, currently not indexed. A content judgement.
The trap where the two interact
Blocking a URL in robots.txt makes it uncrawlable, which means any noindex on it can never be seen. If the page was already indexed, it can stay indexed indefinitely, usually with no description.
- To remove an indexed page: keep it crawlable, serve noindex, wait for it to drop, then block if you wish.
- To prevent a never-indexed URL being crawled: robots.txt alone.
- Never both simultaneously on a page you want removed.
This single interaction accounts for a large share of pages that are stuck in an unwanted state. See robots.txt blocking Google.
Why the distinction saves time
Most wasted effort in this area comes from applying an indexability remedy to a crawlability problem. Rewriting content will not help a page blocked by robots.txt. Improving internal links will not help a page carrying a noindex. Establish the stage first and the correct action is usually obvious.
Frequently asked questions
Can a page be indexed without being crawled?
Occasionally, where it is heavily linked and blocked from crawling. It appears with no description because nothing could be read. This is a symptom of the robots.txt trap rather than a useful state.
Does crawlable mean it will be indexed?
No. Crawling is permission to look; indexing is a decision to include. Many crawlable, technically indexable pages are not indexed because they add nothing new.
How do I check which is failing?
URL Inspection reports both: whether the page was fetched and what the index decision was. The Page indexing report groups the same information across the site.
Should every page be indexable?
No. Filters, internal search, cart and account pages should be excluded deliberately. A site where everything is indexable usually has an index bloat problem. See index bloat.
Consulting CTA
If pages are missing from the index and you cannot tell which stage is failing, book an SEO consultation for a crawl and indexation review.