AI Vocabulary · 07 When It Goes Wrong

What is Data Poisoning?

Planting manipulated data in a model's training or search sources to change how it behaves.

Advanced

Data poisoning means slipping manipulated examples into the data a model learns from or looks things up in. The goal can be a hidden backdoor that stays quiet until a trigger word appears, a bias, or simply worse answers. Poison can enter during pre-training, fine-tuning or in the documents and embeddings behind a search system.

Very little poison can be enough. In a 2025 study by Anthropic, the UK AI Security Institute and the Alan Turing Institute, 250 manipulated documents, about 0.00016% of the training tokens, planted a backdoor in models from 600 million to 13 billion parameters. The backdoor only made models output gibberish, and whether the result holds for larger models or more harmful behaviour is still open.

Why it matters for you

If you fine-tune on scraped or bought data, or let a RAG system read documents from outside, you inherit whatever is hidden in them.

Own it

Know where your training and search data come from, version your datasets and check who can add documents to your knowledge base. Vet data vendors as you would any supplier.

Related