A crawlability audit is a technical SEO review that checks whether search engines can find, access and follow the revenue-driving pages your website depends on. It is used when service pages, ecommerce categories, product collections, location pages or other search-led landing pages are live, but are not being crawled, indexed or surfaced as expected.
The issue is often not that the page is missing or that the content is automatically weak. The problem may be that Google cannot reach the page cleanly, the site structure does not support it, another URL is being treated as the preferred version, or low-value URLs are competing for attention.
SEO Strategist uses a crawlability audit to find where that breakdown is happening. The audit shows whether search engines can move through the site in the same order your business expects: from hubs, to service pages, to categories, to supporting content, and finally to the pages most likely to drive enquiries or revenue.
For South African service and ecommerce businesses, this is especially important when a small number of category, service or location pages carry most of the site’s enquiry or sales potential.
For a broader review of technical, content and visibility issues, see our SEO diagnostic audit.
When a crawlability audit is the right starting point
A crawlability audit is the right place to start when the problem looks technical, structural or indexation-related rather than purely content-related.
For example, a service business may have well-written pages for its main offerings, but those pages sit several clicks away from the homepage and are not linked from the main service hub. An ecommerce site may rely on category pages for qualified search demand, but Google spends more time discovering filtered URLs, parameter URLs and duplicate product listing pages. A business that has recently redesigned its website may find that old URLs still appear in search data while new service or category URLs take too long to settle.
In each case, publishing more content may not fix the underlying issue. Before investing in more pages, link acquisition or redesign work, the business needs to know whether search engines can actually reach and process the URLs that carry commercial value.
You may need a crawlability audit if:
- key service, category, product or location pages are missing from Google’s index
- new URLs take too long to appear in search results
- Google Search Console shows “discovered” or “crawled” URLs that are not indexed
- low-value URLs appear in search data while stronger pages are ignored
- canonical tags, redirects or noindex rules do not match the intended SEO structure
- old, duplicate, filtered or parameter URLs are being crawled more than they should be
- valuable landing pages are buried too deeply in the site architecture
These symptoms need careful diagnosis. Changing robots.txt, canonicals, noindex tags, redirects or sitemap rules without understanding the cause can make the site harder for search engines to interpret.
Crawlability audit vs indexation audit vs technical SEO audit
Crawlability, indexation and technical SEO are closely related, but they answer different questions.
A crawlability audit asks whether search engines can reach and follow the right URLs. An indexation audit asks whether those URLs are being included, excluded or canonicalised correctly. A technical SEO audit is wider and may also cover performance, rendering, structured data, redirects, mobile usability and broader site checks.
| Audit type | Main question | Best fit |
|---|---|---|
| Crawlability audit | Can search engines find, access and follow the URLs that matter? | Service pages, category pages or product collections are buried, blocked, orphaned or difficult to reach. |
| Indexation audit | Are the right pages being indexed, excluded or canonicalised? | Pages are discovered but not indexed, excluded unexpectedly or indexed under the wrong URL. |
| Technical SEO audit | What technical issues are affecting how the site is crawled, rendered and processed? | The site needs a broader technical review across crawling, indexing, speed, rendering, redirects and structured data. |
| SEO diagnostic audit | What is holding back the site’s ability to attract qualified enquiries from search? | The business needs a wider review of technical issues, content, architecture, keyword targeting and commercial priorities. |
In practice, these issues often overlap. A page may fail to appear in search because it is blocked, poorly linked, canonicalised elsewhere, duplicated, thin, or treated as less important than other URLs in the site structure. The value of the audit is in identifying which cause applies before implementation begins.
For wider technical issues, see our website technical audit.
What the audit checks
A crawlability audit reviews the routes and rules search engines encounter when they move through your site. Google describes crawling and indexing as separate stages of Search: crawling is how Google discovers pages, while indexing is how it analyses and stores eligible pages for possible search results. (developers.google.com)
The audit starts with access. It checks whether crawlers are allowed to reach the URLs that should support qualified search demand. This includes robots.txt rules, meta robots tags, X-Robots-Tag headers, blocked resources, server responses and redirect behaviour. Google’s robots.txt guidance states that robots.txt controls crawler access, but it is not a reliable way to keep a page out of Google’s index; noindex or access control is the right route when the goal is to prevent indexing. (developers.google.com)
The next layer is internal discovery. A page should not rely only on an XML sitemap to be found. It should also be linked from relevant hubs, menus, category pages, service pages or supporting content. Google’s link guidance explains that links help Google find pages and understand page relevance. (developers.google.com)
The audit then checks URL preference. Search engines need consistent signals about which version of a page should be treated as the main version. Canonical tags, redirects, internal links and sitemap inclusion should not point in conflicting directions. Google’s canonicalisation guidance explains that redirects and rel="canonical" annotations are strong canonical signals, while sitemap inclusion is a weaker signal that can still support the preferred version. (developers.google.com)
The final layer is crawl waste. Larger websites often create unnecessary crawl paths through filters, tags, internal search pages, parameters, duplicate templates or old URLs. This is especially common on ecommerce and directory-style sites. Google’s URL structure guidance warns that some dynamic URL patterns, including filtering and search-result URLs, can cause inefficient crawling if not handled carefully. (developers.google.com)
Example: category pages are live, but Google is not indexing them
A common crawlability problem is an ecommerce category page that exists in the CMS, appears in the sitemap and can be opened by users, but still does not perform in search.
At first glance, this may look like a content issue. The page may need better copy, stronger product coverage or clearer targeting. But the audit may show a deeper problem: the category is not linked from the main shop structure, filtered versions of the category are crawlable, the canonical tag points to a different URL, and the XML sitemap includes both the main category and several low-value variants.
Before the audit, the business sees the issue as: “Google is not ranking this page.”
After the audit, the issue becomes more specific: Google is receiving mixed signals about which category URL matters, and the site architecture is not supporting the main version strongly enough.
The fix path may include strengthening internal links to the main category, cleaning sitemap entries, reviewing canonical rules, reducing crawlable filter combinations and improving the category page itself once the technical and structural signals are clear.
That is the difference between guessing and diagnosing.
How findings are prioritised
A useful crawlability audit does not hand over a long spreadsheet of errors and leave the business to work out the order. The value is in deciding which fixes protect high-value URLs, which fixes need development time, and which issues are safe to handle later.
SEO Strategist prioritises findings by business impact, technical severity and implementation complexity.
| Priority | Meaning | Example |
|---|---|---|
| Critical | The issue may stop a valuable URL from being crawled or indexed correctly. | A noindex tag is present on a main service page. |
| High | The issue weakens discovery or URL preference for a key page group. | Core category pages are not linked from relevant hubs. |
| Medium | The issue creates crawl inefficiency, duplication or mixed signals. | XML sitemaps include redirected, blocked or low-value URLs. |
| Low | The issue should be cleaned up, but is unlikely to be the first SEO bottleneck. | A minor broken link appears on an old support page. |
This matters because crawlability work often involves more than one team. A sitemap clean-up may sit with the SEO team or CMS owner. Redirect changes may need a developer. Navigation and internal-link changes may involve marketing, UX and content. Canonical or filter-handling decisions on ecommerce sites may need both SEO and development input.
Sample audit finding
A crawlability audit should make the issue clear enough for the right person to act on it.
| Finding | Main ecommerce category pages are not linked from the primary shop hub. |
|---|---|
| Severity | High |
| Affected URLs | /shop/category-a/, /shop/category-b/, /shop/category-c/ |
| Why it matters | These pages rely on the sitemap and product links for discovery, but the site architecture does not clearly identify them as core category pages. |
| Recommended fix | Add crawlable links from the shop hub and relevant parent categories using descriptive anchor text. |
| Owner | SEO/content team for link placement; developer if template changes are required. |
| Next check | Re-crawl the category path and review Google Search Console discovery/indexation changes after implementation. |
This is the level of clarity the audit should provide: not only what is wrong, but where it happens, why it matters, who should handle it, and how it should be checked after the fix.
What you receive
The audit output is designed to help your team make decisions, not just collect technical observations.
For each meaningful issue, the audit explains what is happening, why it matters, which URLs are affected, how serious it is, who likely needs to fix it, and what action should follow.
A typical audit may cover crawl access problems, noindex or canonical conflicts, sitemap quality, redirect issues, orphaned URLs, excessive crawl depth, weak internal linking, duplicate URL patterns, ecommerce filter issues and Search Console indexation patterns.
The outcome is a clearer implementation path: which URLs need stronger internal links, which technical rules need changing, which pages should be consolidated, which low-value URL patterns need controlling, and which findings should be checked again after implementation.
Google notes that sitemaps can help it crawl a site more efficiently, but submitting a sitemap is still a hint and does not guarantee that every URL will be crawled or used for crawling. (developers.google.com)
What happens after the audit
After the audit, the next step is implementation.
Some fixes are direct. A noindex tag can be removed, a sitemap can be cleaned, a broken internal link can be corrected, or a redirect can be updated. Other fixes need sequencing. If an ecommerce site has thousands of crawlable filter URLs, the solution may involve canonical rules, internal-link changes, parameter handling, category-page improvements and developer input.
The audit helps prevent a common SEO mistake: spending money on new content or new links while the site still struggles to expose its existing service, category or product pages properly.
For businesses that need sequencing after the diagnostic work, the audit can connect into an SEO audit roadmap.
Which SEO diagnostic should you choose?
Choose a crawlability audit when the main question is: can search engines reach, follow and process the pages your business needs to be visible?
Choose a broader SEO diagnostic audit when the issue is less clear. If traffic has dropped, rankings are inconsistent, service pages are underperforming, content is poorly targeted and the site has technical concerns, the problem may need a wider review than crawlability alone.
Choose a website technical audit when the site has broader technical issues. That may include crawlability and indexation, but also performance, rendering, redirects, structured data and site-wide technical checks.
SEO Strategist’s SEO diagnostic services page can help you compare the available audit options if the starting point is not yet clear.
Book an SEO diagnostic review
Before investing in more SEO content, link acquisition or development work, confirm that search engines can actually reach, follow and process the pages your business depends on.
A crawlability audit shows whether valuable URLs are blocked, buried, duplicated, weakly linked, canonicalised incorrectly or competing with low-value URL patterns.
SEO Strategist reviews crawlability as part of the wider commercial SEO architecture, not as an isolated technical checklist.