AI Vocabulary · 07 When It Goes Wrong
What is Automation Bias?
The human tendency to trust what a machine says over our own judgement, even when it is wrong.
Automation bias is a human problem, not a technical one. When a system produces a confident answer, people tend to accept it, check less carefully and overlook contradicting evidence. It was studied long before generative AI, for example in aviation and medicine.
With AI it is especially tempting: answers are fluent, fast and usually right, so reviewers slowly stop reading closely. That is exactly when a hallucination slips through. A human in the loop only helps if the human is actually paying attention.
Why it matters for you
Rubber-stamped AI output carries your name. In decisions about money, people or customers, automation bias turns a model's error into your error.
Own it
Make reviewing easier and more deliberate: show sources, highlight uncertain parts, keep review batches small, and sometimes ask reviewers to explain why they agree. Measuring error rates with evals tells people where to look harder.
Where it comes from
The term comes from aviation psychology. Kathleen Mosier, Linda Skitka and colleagues studied pilots in highly automated "glass cockpits" and described automation bias as using automated cues instead of checking information yourself (NASA, 1996/97). A detail worth keeping: pilots who felt accountable for their decisions cross-checked the automation more and made fewer errors. That lesson carries straight over to AI.
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.