AI Vocabulary · 04 Building with AI

What is Observability?

Systematically recording what an AI system does, every model call, tool call and result, so you can see why it acted as it did.

Intermediate

Observability means collecting traces: complete records of a run with every model call, every tool call, every result, plus time and cost. For an agent this matters more than for classic software, because much of its logic sits in the model and changes from run to run.

Traces are the raw material for quality work. Harrison Chase of the framework vendor LangChain describes understanding agents by reading traces and judging quality by having people label them. Anthropic calls the trace of a test run a transcript and uses it in evals. For the format, the MCP specification recommends the open standard OpenTelemetry.

Why it matters for you

When a customer complains, a trace shows what the agent saw, which tools it called and where it went wrong. Without traces you can only guess, and you cannot prove what happened.

Own it

Log every agent run from day one, in an open format you can take to another tool. Check who can read the traces, since they often contain customer data.

Related