AI Vocabulary · 07 When It Goes Wrong
What is Memorisation?
When a model stores pieces of its training data and can repeat them word for word, including personal data.
Models mostly learn patterns, but they also store some passages of their training data verbatim. This is memorisation. Researchers extracted hundreds of verbatim sequences from GPT-2, including names, phone numbers and email addresses, some of which appeared in only one document. In 2023 a team extracted gigabytes of training data from open and closed models and found that alignment training did not remove memorisation.
Memorisation has legal consequences. A Munich court ruled in November 2025 that song lyrics memorised in a model are a reproduction under copyright law (GEMA v OpenAI, not final). Under the GDPR, European regulators say a model can only count as anonymous if personal data cannot be extracted from it, judged case by case.
Why it matters for you
Anything that goes into training can come out again. Once trained in, data is practically impossible to remove from a model.
Own it
Keep secrets, passwords and personal data out of any data that might be used for training, including chats on consumer tiers that train by default. Remove duplicates from your own fine-tuning data, since repeated text is memorised more easily.
Where it comes from
The production-model results are in Scalable Extraction of Training Data from (Production) Language Models by Nasr, Carlini and colleagues (2023).
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.