AI Vocabulary · 01 The Model

What is Web Crawl?

Automatically downloading huge numbers of public web pages. The raw material of almost every training dataset.

Basic

A web crawler is a program that visits public web pages, saves them and follows their links to find more. A web crawl is the collection it produces. The best-known open archive is Common Crawl: its October 2026 crawl alone holds 2.16 billion pages and 371 TiB of uncompressed data.

Most training data for language models starts as a crawl like this. Before training, developers extract the text, filter out spam and duplicates and keep only a small, high-quality share. Website owners can ask crawlers to stay away with a file called robots.txt, which crawlers follow voluntarily.

Why it matters for you

Your public website, blog and documentation are probably already in several crawls and may have shaped models you use. Whether that is allowed depends on the region and on the signals you set.

Own it

Decide which crawlers may read your site and record it in your robots.txt. Check what you publish: anything public can end up in training data.

Related