Crawling

Before a search engine can show your page, it first needs to know that the page exists and what is on it. That happens through crawling: automated programs, called crawlers or bots, move from link to link and fetch the content of pages. Googlebot is the best known, but Bing and the AI companies behind ChatGPT, Perplexity and other tools also send out their own crawlers.

Crawling is the first step in a sequence. After fetching comes processing and possibly inclusion in the index, and only then can a page appear in search results or in an AI answer. If a page is never crawled, it essentially stays invisible to those systems.

How does a crawler find your pages?

Crawlers discover new URLs in a few ways:

• Through links: from pages they already know, they follow links to other pages, both within your own site and from other sites.
• Through your sitemap: an XML file listing your most important URLs.
• Through manual requests: in Google Search Console, you can have a URL inspected and request indexing.

A page that no link points to, known as an orphan page, is therefore hard to find. Good internal links are essential.

What determines whether a crawler can read your content?

Being found is one thing; being read is another. A few factors play a part:

• Your robots.txt tells crawlers which parts of your site they may not visit. One wrong rule can accidentally shut off your entire site.
• Server errors and slow responses can cause crawlers to fetch fewer pages.
• JavaScript: Google can render JavaScript, but sometimes does so later. Many AI crawlers probably run little or no JavaScript, so content that only appears via scripts may be invisible to them.
• Logins and forms: crawlers can't reach content behind a login.

Crawling by AI bots

More and more businesses are wondering whether they should allow AI crawlers. Some bots collect data to train language models; others fetch pages live to answer a user's question. If you block that second group, you reduce the chance that ChatGPT Search or Perplexity uses your page as a source. So it is a conscious trade-off. The llms.txt file is a more recent proposal for giving AI systems an overview of your most important content, although support for it is still limited.

An example: a new website for a dental practice

A dental practice in Leeds launches a new website. During development, the site was rightly hidden from search engines, but that block in robots.txt was left in place at launch. Weeks later, the site still can't be found in Google. After the rule is removed and the sitemap is submitted in Search Console, the pages are crawled again. It is a common mistake that you can avoid with a simple check after every launch.

How to check your site is being crawled properly

1. In Google Search Console, look at the crawl stats report and the status of your pages.
2. Test important URLs with the URL Inspection tool.
3. Check your robots.txt for unintended blocks, including for AI crawlers.
4. Look for orphan pages and broken links.
5. Check that your main text is in the HTML and isn't only loaded via JavaScript.

For large sites, crawl budget also comes into play. If you want this checked thoroughly, a technical SEO analysis gives you a clear overview of crawl issues and blocks.