AI Vocabulary · 01 The Model
What is Deduplication?
Removing copies and near-copies from training data, so a model learns more and memorises less.
The web is full of repetition: boilerplate, mirrored pages, copied articles. Deduplication finds and removes these copies before training. Exact copies are easy to spot; for near-copies, developers use fast fingerprinting methods such as MinHash. Meta's Llama 3 removes duplicates at three levels: web addresses, whole documents and single lines that repeat too often.
The effect is large. In the older C4 dataset one 61-word sentence appeared more than 60,000 times. Removing duplicates cut verbatim memorisation about tenfold and reached the same accuracy in fewer training steps. It also helps keep test questions out of training data.
Why it matters for you
Duplicated data wastes compute and raises the risk that a model repeats text word for word, including personal data. The same applies to your own datasets for fine-tuning or search.
Own it
Before you fine-tune or build a search index on your documents, remove duplicate files, old versions and copied templates. Keep one authoritative version of each document.
Where it comes from
The tenfold effect and the C4 example are from Deduplicating Training Data Makes Language Models Better by Lee and colleagues (2021).
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.