A noindex directive tells search engines to keep a page out of the index. Used deliberately it is a precise and useful instrument. Applied accidentally it removes pages, sections or entire sites from search results, and it is one of the few SEO faults that can do that overnight.
The failure that matters most
The single most damaging version is a staging-site noindex shipped to production. Development environments are routinely set to noindex, and when the setting travels with the deployment, the live site instructs search engines to remove everything.
- In WordPress, the Search Engine Visibility setting under Reading does this site-wide.
- Theme or plugin settings can apply it per template or per post type.
- Server or CDN rules can emit an
X-Robots-Tagheader that nothing in the page source reveals. - Build configurations can inject the tag per environment and be misconfigured.
Where traffic falls sharply and broadly after a deployment, this is the first thing to check, ahead of any content or algorithmic explanation. It is covered as a diagnostic step under SEO recovery.
The header nobody checks
Noindex can be delivered two ways: a meta tag in the HTML, or an X-Robots-Tag HTTP header. The header is invisible when viewing page source, which is why this fault sometimes survives weeks of investigation.
- Check response headers directly, not just the rendered HTML.
- URL Inspection reports the directive regardless of delivery method, which makes it the reliable check.
- CDN and edge configurations can add the header without the origin server knowing.
Where noindex is correct
Not every noindex is a fault. Applied deliberately it is often the right answer.
- Internal search result pages.
- Filter and sort URLs with no standalone demand.
- Thank-you, confirmation and account pages.
- Thin tag or archive pages that duplicate main listings.
- Duplicate print or export views.
The interaction with robots.txt that catches people out
Blocking a URL in robots.txt prevents crawling. If the page is already indexed, the crawler can no longer fetch it, cannot see the noindex, and the page may remain indexed indefinitely.
- To remove an indexed page: allow crawling and serve noindex, then wait for it to drop.
- To keep a never-indexed page from being crawled: robots.txt is appropriate.
- Never apply both at once to a page you want removed. They work against each other.
- Once the page has dropped from the index, robots.txt can be applied to save crawl capacity.
Auditing it
- Crawl the site and list every URL carrying a noindex directive from either source.
- Compare that list against what you intended. The gap is your problem set.
- Check the WordPress Reading setting after any migration or restore.
- Verify headers on a sample of templates, not only the homepage.
- Re-check after every deployment, since this fault recurs.
Frequently asked questions
How quickly does noindex take effect?
As soon as the page is recrawled, which for important pages can be within days. That speed is why an accidental sitewide noindex is so damaging and why it should be checked first after a sharp decline.
How long does recovery take after removing it?
Pages return as they are recrawled, typically over days to weeks depending on crawl frequency. Important pages usually return quickly; deeper pages take longer.
Should I use noindex or robots.txt?
Noindex to remove something already indexed. Robots.txt to prevent crawling of URLs that were never indexed. Using robots.txt on an indexed page can leave it stuck in the index.
Does noindex pass link equity?
A noindexed page can still be crawled and its links followed, but over time such pages tend to be crawled less. Do not rely on noindexed pages as a route for internal linking.
Consulting CTA
If pages have disappeared from search after a deployment or migration, book an SEO consultation to check indexation directives before anything else is changed.