AI Vocabulary · 01 The Model
What is Training Data?
The text, images, code and other examples a model learns from. What goes in shapes what comes out.
Training data is everything a model sees during training. For language models, most of it comes from the public web, plus books, code and research papers. The open Dolma corpus, released in 2024, gives a sense of scale: three trillion tokens from web pages, papers, code, public-domain books, social media and encyclopedias.
The supply is not endless. In 2024, the research institute Epoch AI estimated the stock of public human-written text at around 300 trillion tokens, with a wide range of 100 to 1,000 trillion, and expected models to use it up sometime between 2026 and 2032.
Access is also shrinking. A 2024 study of 14,000 web domains found that more than 5 percent of all tokens in C4, a widely used training set, and more than 28 percent of its most actively maintained sources, had become fully restricted by the websites' own rules.
Why it matters for you
A model can only be as good, and as fair, as the data behind it. Gaps and bias in the data become gaps and bias in the answers. For most commercial models you cannot see what went in, so ask about sources, licences and copyright.
Own it
Your documents, decisions and customer conversations are training data nobody else has. Keep them clean, documented and under your rights. That is the practical side of data sovereignty.
Where it comes from
"Training data" is general machine learning vocabulary without a single inventor. The figures above come from Dolma by Luca Soldaini and colleagues (2024), Will we run out of data? by Pablo Villalobos and colleagues at Epoch AI (2022, updated June 2024) and Consent in Crisis by Shayne Longpre and colleagues (2024).
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.