Crawl budget

Search engines don't visit your website without limits. Googlebot decides for itself how many pages it fetches in a given period, based on how much your server can handle and how interesting or frequently updated your pages seem. That amount of attention is what we call crawl budget. It partly determines how quickly new or updated pages get noticed.

For most small websites this isn't a big issue. Google itself says crawl budget mainly becomes relevant for large sites with tens of thousands of pages or sites that add new content very quickly. Still, it's useful to understand the principle, especially as your site grows or generates lots of pages automatically.

How is crawl budget determined?

Google describes two components:

• Crawl capacity: how many requests your server can handle without slowing down. If your site responds slowly or returns errors, Google scales back the pace.
• Crawl demand: how much Google wants to fetch your pages. Popular, regularly updated pages are visited more often than outdated or poorly linked pages.

Both components change constantly: a faster server or more popular content can increase the budget, while errors reduce it. The result is a practical limit on how often and how deeply Googlebot crawls your site.

When should you be concerned?

Crawl budget becomes a point of attention when you notice that important pages take a long time to be fetched or indexed. Typical situations:

• an online shop with lots of filter combinations that each produce a separate URL;
• a site that creates hundreds or thousands of pages through programmatic SEO;
• lots of duplicate content, for example due to parameters in URLs or print versions;
• long redirect chains or lots of broken links.

How to use your crawl budget efficiently

1. Keep your sitemap clean: only pages you genuinely want indexed.
2. Limit unnecessary URLs such as endless filter combinations or session parameters, for example with a targeted rule in robots.txt.
3. Point duplicate variants to the main version with a canonical tag.
4. Fix errors and redirect chains, so crawlers don't waste time.
5. Make sure your server is fast: a site that responds quickly can handle more requests.
6. Link internally to your most important pages, so they're recognised as a priority.

An example: an estate agency with lots of pages

An estate agency in the Manchester area publishes a separate page for every property and keeps sold properties online. On top of that, its search function generates thousands of URLs with combinations of price, town and property type. As a result, new properties sometimes only appear in Google after several days. By excluding the filter URLs, handling sold properties in a structured way and providing an up-to-date sitemap, the agency helps Googlebot focus its attention on the new listings.

Crawl budget and AI crawlers

Besides Googlebot, crawlers from AI companies, such as those of OpenAI or Perplexity, also visit your site. They have limited resources too and probably mainly fetch pages that are easy to reach and clearly structured. So a tidy site helps both classic search engines and AI tools. If you build lots of pages at scale, take a look at how the programmatic SEO feature takes a healthy structure and indexability into account.