AI Vocabulary · 08 Rules & Responsibility

What is robots.txt?

A small text file on a website that tells crawlers, including AI crawlers, which pages they should skip.

Basic

robots.txt is a plain text file at the root of a website, standardised as RFC 9309. It lists crawlers by name, such as GPTBot, ClaudeBot or CCBot, and says which parts of the site each may visit. Crawlers follow it voluntarily; it is a request, and a firewall is a separate matter.

How many sites block AI crawlers depends on the sample: about 6% of some 140 million websites (Ahrefs, an SEO vendor) and over 28% of the most actively maintained sources behind the C4 training dataset (Longpre and colleagues, 2024). The EU's code of practice for general-purpose AI asks signatories to respect robots.txt. At the IETF, a vocabulary that separates AI training, use as model input and search is still a draft.

Why it matters for you

Your robots.txt is the simplest signal you can send about how AI companies may use your public content. In the EU it can count as a machine-readable reservation of rights.

Own it

Open yoursite.com/robots.txt and check what it says about AI crawlers. Decide per crawler whether you want to be found in AI search, used for training, or neither.

Related