robots.txt controls what crawlers may request. Used correctly it saves crawl capacity. Used carelessly it removes a site from search, and it is one of the few files where a single line can do that.
The most important thing to understand about it is what it does not do: it does not remove pages from the index.
The misconception that causes most damage
Blocking a URL in robots.txt prevents crawling. It does not deindex. If the page is already indexed, the crawler can no longer fetch it, cannot see any noindex directive you add, and the page can remain in the index indefinitely, often shown with no description.
- To remove an indexed page: allow crawling and serve a noindex, then wait for it to drop.
- To prevent crawling of URLs never indexed: robots.txt is the right tool.
- Never both at once on a page you want removed. They work against each other.
- After the page has dropped, you can then block it to save crawl capacity.
Getting this backwards is common and produces pages that are stuck: not crawlable, not removable. See noindex tag issues.
The rules that block a site by accident
Disallow: /blocks everything. Almost always carried over from staging.Disallow: /wp-content/can block CSS and JavaScript, so pages render incorrectly when assessed.- A rule under
User-agent: *intended for one bot but applied to all. - Blocking a path that also contains pages you need, since matching is by prefix.
- A staging robots.txt deployed with the site, which is the single most common version of this fault.
Where traffic falls sharply and broadly after a deployment, read robots.txt before anything else. It takes seconds and rules out the fastest-acting cause. This is also a first step in SEO recovery.
What it is genuinely good for
- Preventing crawling of filter and parameter URLs before they are ever indexed.
- Keeping crawlers out of internal search results, which can be generated endlessly.
- Excluding cart, checkout and account paths.
- Protecting genuinely infinite spaces such as calendars.
- Recovering crawl capacity on large catalogues. See crawl budget.
Note the pattern: it is a preventative tool, applied before URLs enter the index, not a cleanup tool applied after.
Checking yours
- Read the live file at
yourdomain.co.za/robots.txtrather than a plugin’s summary of it. - Confirm CSS and JavaScript required for rendering are not blocked.
- Use URL Inspection with a live test on a blocked URL to see the actual effect.
- Check it again after every deployment, since this fault recurs.
- Confirm the sitemap is referenced in it.
Frequently asked questions
Will blocking a page remove it from Google?
No, and this is the core misunderstanding. Blocking prevents crawling. An already-indexed page can stay indexed, often with no description shown, because the crawler cannot see any directive telling it otherwise.
Should I block filter URLs?
Yes, if they were never indexed. If they are already indexed, allow crawling with a noindex first, let them drop, then block. Blocking first leaves them stuck.
Can robots.txt hurt rankings?
Indirectly and severely. Blocking pages you need crawled, or blocking resources required to render them, both cause real damage. The file itself is not a ranking factor; what it prevents is.
Should I block CSS and JavaScript?
No. Google renders pages and needs those files to assess them properly. Blocking them is a legacy practice that now causes harm.
Consulting CTA
If pages you need indexed are blocked, or blocked pages are stuck in the index, book an SEO consultation for a crawl directive review.