AI Vocabulary · 07 When It Goes Wrong

What is Model Collapse?

What happens when models are trained again and again on AI-generated data: the rare cases disappear, and the damage stays.

Advanced

More and more text, images and code online are made by AI. If the next generation of models is trained on that output, and the one after on its output, something gets lost. Ilia Shumailov and colleagues showed in 2023 that the rare cases at the edges of the original data, the "tails", disappear first, and that these defects cannot be undone.

They observed it in several kinds of models, including language models, and called it model collapse. The authors concluded that data from genuine human interaction becomes more valuable as AI-generated content spreads.

The finding is about repeated training on generated output. It does not mean that every use of synthetic data is harmful: a curated mix with real data is a different situation.

Why it matters for you

The rare cases are often the ones that matter most: unusual customers, minority languages, edge cases in your process. A model that has lost them answers confidently and generically, and nobody notices.

Own it

Your real, human-made data is becoming scarcer and more valuable. Keep it, document where it comes from, and keep it separate from AI-generated material. Test your systems on real rare cases with evals.

Related