robots.txt is a small file with disproportionate consequences. A single stray line can remove a site from search, and because nothing visibly breaks, it can sit there for months.
This page is about auditing yours. For the distinction between blocking and deindexing — the most common misuse — see robots.txt vs noindex.
What it does
It tells crawlers which paths they may fetch. That is the whole scope. It does not control indexing, it does not secure anything, and it is publicly readable at /robots.txt on every site including yours.
Auditing it
1. Read the live file
Fetch it directly rather than trusting a plugin’s preview. What is served is what counts, and staging files, caching layers and CDN rules can all produce a difference.
2. Look for the catastrophic line
Disallow: / blocks the entire site. It belongs on staging and ships to production more often than anyone would like. If traffic fell sharply after a launch, check this first.
3. Check what each rule actually matches
Rules match path prefixes, so Disallow: /admin also blocks /administrator-guide/. Broad prefixes catch more than intended, and the damage is invisible until you look.
4. Confirm nothing important is blocked
- CSS and JavaScript needed to render pages. Blocking these means pages are assessed on an incomplete rendering.
- Image directories, if image search matters to you.
- Any section generating revenue.
5. Confirm the sitemap is declared
A Sitemap: line pointing at the live sitemap URL. Cheap to add, easy to leave stale after a migration. See XML sitemaps.
6. Check for pages you are trying to deindex
Anything blocked here cannot be crawled, so any noindex on it is unreachable. Blocked pages you want removed will stay.
What belongs in it
- Genuine crawl waste at scale — filter and sort parameters, internal search results, session URLs.
- Administrative and system paths with no public value.
- The sitemap declaration.
On a large store, blocking parameter URLs is legitimate and valuable. See crawl budget.
What does not belong in it
- Pages you want removed from results. Use noindex.
- Anything private. The file is a public list of paths.
- Rendering resources.
- noindex directives. Unsupported and ignored.
Frequently asked questions
Do I need one at all?
Not strictly. A small site with nothing to exclude functions fine without one, and having a minimal file with the sitemap declared is still worthwhile.
How do I test a rule?
Use the robots.txt tester in Search Console against specific URLs, rather than reasoning about prefixes.
Does blocking save crawl budget?
At scale, yes. On a small site there is nothing meaningful to save.
My file looks right but pages are still blocked.
Check what is actually served — caching or a CDN may be returning an older version. See robots.txt blocking Google.
Consulting CTA
If you are unsure what your robots.txt is excluding, book an SEO consultation.