Indexing

Indexing is the step in which a search engine analyses a page it has visited and stores it in its database, the index. Only once a page is in that index can it appear in the search results. A page that isn't indexed effectively doesn't exist for Google, however good the content is.

Indexing follows crawling, the discovery and fetching of pages. But not everything that gets crawled ends up in the index. Google decides for itself which pages it considers valuable enough to include, and says it doesn't index every page of every site.

How does indexing work?

After Googlebot has fetched a page, a series of steps follows. Google renders the page, roughly the way a browser does, and looks at the text, images, links and structured data. It then determines whether the page is a duplicate of another one and which version is the canonical. Finally, it decides whether the page goes into the index and which topics it seems relevant for.

Why isn't a page indexed?

In Google Search Console you'll find the reasons in the Pages report. Common causes are:

• A noindex instruction, sometimes accidentally left in place after a new site launches.
• A block in robots.txt that stops Google from fetching the page.
• The page is seen as a duplicate and another URL has been chosen as canonical.
• "Crawled – currently not indexed": Google has looked at the page but didn't (yet) find it valuable enough. This is often thin or very similar content.
• "Discovered – currently not indexed": Google knows the URL but hasn't fetched it yet.
• Server errors or pages that are only visible after logging in.

An example: the online store with thousands of filter pages

A sports equipment store in Leeds has 800 products, but filters for size, colour and brand create tens of thousands of URLs. Search Console shows that a large share of them aren't indexed, while some new product pages wait weeks to be included. The solution lies in focus: excluding filter combinations with no search value of their own, setting canonicals correctly and limiting the sitemap to pages that really belong in the index. That way, Google's attention goes to the pages that count.

Indexing and AI search

Google AI Overviews and AI Mode build on the Google Search index. What isn't in it can't appear there as a source. Other AI search engines, such as ChatGPT Search and Perplexity, use their own crawlers and often external search indexes such as Bing's as well. Good indexing across several search engines is therefore a basic requirement for AI visibility. Also check that your robots.txt isn't unintentionally blocking the crawlers of AI services.

How to keep indexing under control

1. Submit an up-to-date sitemap containing only indexable pages.
2. Check the Pages report every month for new exclusions.
3. Improve or merge thin pages instead of endlessly resubmitting them.
4. Use URL Inspection after important changes.

For larger sites, a technical SEO analysis that tracks down indexing problems pays off before they cost you traffic.