AI Vocabulary · 07 When It Goes Wrong
What is Red Teaming?
Deliberately attacking your own AI system, the way an adversary would, to find weaknesses before others do.
In red teaming, people, or other AI systems, try on purpose to make an AI system misbehave: to produce harmful content, leak data, ignore its rules or fall for prompt injection and jailbreaks. What they find is used to fix the system before it reaches users.
The term comes from security. The US standards institute NIST defines a red team as a group authorised and organised to emulate a potential adversary's attack capabilities. For AI, there is no single agreed definition yet. NIST's 2024 profile for generative AI recommends AI red-teaming among its risk measures.
In a 2022 study, Anthropic's researchers ran 38,961 red-team attacks on their own language models and published them as a dataset. Models trained with human feedback became harder to attack as they grew larger. Anthropic sells such models.
The EU AI Act makes it a duty for some: providers of general-purpose models with systemic risk must evaluate them with state-of-the-art methods, including conducting and documenting adversarial testing (Article 55).
Why it matters for you
Anything you build on AI, a chatbot on your website, an agent with access to your systems, will be tested by real users, some with bad intentions. Better that you find the holes first.
Own it
Before launch, spend an afternoon trying to break your own system: odd questions, hostile instructions, hidden commands in documents. Turn what you find into evals and guardrails.
Where it comes from
The security definition is in the NIST glossary, the AI recommendation in NIST AI 600-1 (July 2024). The study is Red Teaming Language Models to Reduce Harms by Deep Ganguli and colleagues (August 2022). The legal duty is in Article 55 of the EU AI Act.
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.