These two are constantly used interchangeably and they do opposite jobs. Choosing the wrong one is one of the most common technical SEO mistakes, and the failure is silent — the page stays in search results and you assume the instruction is working.
What each actually does
- robots.txt controls crawling. It asks search engines not to fetch a URL. It says nothing about indexing.
- noindex controls indexing. It tells search engines not to keep the page in results. To see it, they must crawl the page.
That dependency is the whole point. A page blocked in robots.txt cannot be crawled, so its noindex is never seen. Blocking a page you want removed guarantees it stays.
The mistake, step by step
A page appears in search that shouldn’t. Someone adds a robots.txt disallow. The page remains, sometimes for months, often with the description replaced by a note that no information is available.
What happened: the crawler stopped fetching the page, so it can no longer see any removal instruction. Blocking froze the page in place instead of removing it.
The fix is to reverse it — allow crawling, add noindex, wait for the recrawl, and only then consider blocking if you want to save crawl capacity.
Which to use
Use noindex when
- The page must not appear in search results.
- It is already indexed and you want it gone.
- It has value to visitors but not to searchers — thank-you pages, internal search results, filtered views.
Use robots.txt when
- You want to conserve crawl capacity on URLs that add nothing.
- The volume is large — thousands of parameter or filter URLs.
- The pages are not indexed and you want to keep it that way.
On a large store, blocking filter parameters is legitimate and useful. See crawl budget and URL parameters.
Never combine them on the same URL
Blocking and noindexing together is self-defeating: the block prevents the noindex from being read. If a page must be removed, allow the crawl until it has dropped out, then block it if you still want to.
What neither does
- Neither is security. A robots.txt disallow is a public list of paths you would rather people not visit. Use authentication for anything genuinely private.
- Neither removes a page immediately. Both take a recrawl to take effect.
- Neither stops other sites linking to the URL.
Frequently asked questions
My page is still in Google after adding noindex — why?
Either it has not been recrawled yet, or the URL is also blocked in robots.txt so the directive is unreachable. Check the second first.
How do I remove a page urgently?
Use the removal tool in Search Console for a temporary suppression, and add noindex for the permanent fix. The tool alone expires.
Does blocking pass link equity?
A blocked URL cannot pass anything onward, because the links on it are never seen.
What about noindex in robots.txt?
Unsupported. It was never an official directive and is not obeyed. Use a meta robots tag or an X-Robots-Tag header.
Consulting CTA
If pages you have tried to remove are still appearing, book an SEO consultation. See also robots.txt blocking Google.