Sitemap

A sitemap is an overview of the pages on your website that you want search engines to find. In SEO, this usually means an XML sitemap: a file that isn't meant for visitors, but for crawlers. It lists the URLs of your important pages and often also when they were last changed.

A sitemap is a helpful tool, not a guarantee. Google says a sitemap helps it discover pages, but that it decides for itself which pages it crawls and indexes. A page that is in your sitemap but has little value or is technically blocked won't automatically make it into the search results.

When is a sitemap really useful?

Every website benefits from one, but the difference is greatest in these situations:

• on large websites with lots of pages, such as online stores with hundreds of products;
• on new websites that don't have many external links yet;
• for pages that are poorly linked internally and are therefore hard to find during crawling;
• on sites that publish new content often, such as a news section or jobs page.

What should and shouldn't go in it?

A good sitemap is a selection, not a dump of every URL on your server. Only include pages that:

1. return a 200 status code, so no 404 errors or redirects;
2. are indexable, without a noindex instruction;
3. are the preferred version of a page, in line with your canonical tag;
4. aren't blocked in your robots.txt.

It's better to leave out filter pages, internal search results, thank-you pages and test pages. A sitemap full of conflicting signals makes it harder for search engines to trust what you give them.

How to set up your sitemap properly

Most modern website builders and content management systems generate an XML sitemap automatically. Do check that the settings are correct, though. Add the location of your sitemap to your robots.txt and submit it via Google Search Console. There you can also see how many of the submitted pages have actually been indexed and what issues there are.

An example: an interiors shop in Bristol relaunches its online store. After the migration, the sitemap still contains hundreds of old product links that now redirect. By cleaning up the sitemap so that only the new, live URLs are in it, the shop gives Google a clear overview of what really matters.

For larger websites, you can also split your sitemap up, for example into separate files for products, categories and blog posts, brought together in a sitemap index. That makes it easier to see in Search Console which part of your site has indexing problems. A single sitemap can contain a maximum of 50,000 URLs, by the way.

Sitemaps and AI search engines

AI search environments such as ChatGPT Search and Perplexity also rely on web indexes to find sources. How each system uses sitemaps isn't always public, but a correct sitemap probably makes your site easier to explore for other crawlers too. If you want your latest content to be picked up quickly, it's well worth keeping your sitemap up to date. Not sure whether your sitemap contains errors? A technical SEO analysis shows which URLs are missing or don't belong there.