Robots.txt

Robots.txt is a simple text file in the root folder of your website, available at yourdomain-example.com/robots.txt. It gives crawlers, the bots of search engines and other services, instructions about which parts of your site they may or may not visit. It is usually one of the first files a crawler requests when it visits your site.

Important to know: robots.txt is a request, not a lock on the door. Well-known search engines respect the rules, but the file doesn't secure anything. So you protect sensitive information with a password or other access control, not with robots.txt.

What does a robots.txt file look like?

The file consists of groups of rules, each for a particular crawler. The main elements are:

• User-agent: which bot the rules apply to, for example Googlebot or an asterisk for all bots.
• Disallow: which paths the bot may not crawl, such as a shopping basket or internal search results.
• Allow: exceptions within a blocked folder.
• Sitemap: the location of your sitemap, so crawlers can find your important pages more easily.

Crawling is not the same as indexing

A common misconception is that you keep a page out of Google by blocking it in robots.txt. That isn't quite right. Robots.txt prevents the page from being crawled, but if other sites link to it, the URL can still show up in the search results, albeit without a description. If you really want to keep a page out of indexing, use a noindex instruction and keep the page crawlable, so Google can read that instruction.

Common mistakes

• After a new website launches, a rule blocking the entire site is left in place, carried over from the test environment.
• CSS and JavaScript files are blocked, so Google can't render your page correctly.
• Important folders are accidentally excluded by a rule that is too broad.
• Robots.txt is used to "hide" confidential documents, even though the file itself is publicly readable.

For example, an online store in London notices after a migration that its new product pages aren't appearing in Google. The cause: the test rule "Disallow: /" was still in the live file.

Mistakes like these are easy to prevent. Add robots.txt to your checklist for every launch or migration, check the file after major changes to your website, and use the robots.txt report in Google Search Console to see which version Google has fetched and whether it contains errors. Also keep the file as simple as possible: the fewer rules, the smaller the chance of mistakes.

Robots.txt and AI crawlers

AI companies use crawlers too, such as OpenAI's GPTBot, PerplexityBot or Google-Extended, a separate token that lets you decide whether your content may be used for Google's AI models. Through robots.txt you can choose per bot what you allow. That is a strategic trade-off: if you block AI crawlers completely, you may also reduce your chances of appearing as a source in AI answers. So look at what each bot does before you allow or block it, and bear in mind that not every crawler follows the rules equally strictly.

Not sure whether your robots.txt is blocking important pages? A technical analysis of your website quickly tracks down such blocks and conflicts with your sitemap.