Noindex

With a noindex instruction you ask search engines not to include a page in their search results. The page remains accessible to visitors who know the link or click through from your menu, but it won't appear when someone searches on Google. You usually add the instruction as a meta tag in the page's head (meta name="robots" content="noindex") or via an HTTP header, which is handy for files such as PDFs.

Noindex is a powerful tool for keeping your website tidy in the eyes of search engines. Used incorrectly, however, it can also make important pages invisible.

When is noindex useful?

Not every page on your website belongs in the search results. Typical candidates for noindex are:

• thank-you pages after a form or purchase;
• internal search results and filter combinations in an online shop;
• login and account pages;
• test pages or a staging environment;
• pages with little original content, such as tag archives, that you want to keep for other reasons.

An online shop in Manchester selling kitchenware generates thousands of URLs through combinations of filters such as colour, price and brand. Those pages are useful for visitors, but add little in the search results. By putting noindex on those filter pages, the shop keeps the focus on category and product pages. Sometimes a canonical tag is a better choice, for example when variants are almost identical to one main page.

Noindex versus robots.txt

A common mistake is combining noindex with a block in robots.txt. If you block a page in robots.txt, Google can't crawl it and therefore won't see the noindex tag either. A blocked URL can then still appear in the results, for example when other sites link to it, with or without a description. If you want to keep a page out of the index reliably, leave it crawlable and use noindex.

Mistakes to avoid

1. A noindex that accidentally stays on the whole site after a new website goes live.
2. Noindex on pages that are in your sitemap, which sends conflicting signals.
3. Important service pages that automatically get noindex through a plugin setting.
4. Expecting the page to disappear immediately: Google first has to revisit it before the instruction takes effect.

In Google Search Console, the indexing report shows which pages are excluded by a noindex tag. Check that list regularly. If it contains a page you do want people to find, remove the tag, check that the page is crawlable and request re-indexing via URL Inspection. Even then, it can take some time for the page to reappear in the search results.

Noindex and AI search

Noindex is aimed at search engines. Pages with noindex don't appear in Google's regular search results and therefore probably not as a source in Google AI Overviews either, which rely on the search index. Other AI services use their own crawlers and their own rules. If you want to manage their access, it is best to look at the specific instructions each provider publishes for this.

Not sure whether pages are being wrongly kept out of the index? A technical SEO analysis of your indexing settings quickly tracks down noindex errors like these.