Trace-Based Evals for AI Agents That Actually Work

The Final Answer Is Not Enough Evidence

An agent closes a support ticket with a polished, accurate-sounding reply — and along the way it read a customer record it had no permission to touch. The reply ships. The breach is invisible, because the grader only ever looked at the reply. That is the trap of grading an agent the way you grade a chatbot: read the final answer, decide if it looks right, move on. For a single-turn assistant that can be defensible. For an agent that reads documents, calls tools, and writes to systems, it hides almost everything that matters.

An agent can return a correct-looking answer while having called the wrong tool, skipped a required approval, leaked a permission-protected document, or retried a failing write five times. The text passed. The run did not. For agents, the trace is the unit of review.

This article lays out trace-based evals: what to grade beyond final text, how to structure an eval suite, and a concrete 12-case set you can adapt for a knowledge assistant. The goal is a release loop you can trust instead of a vibe check you hope holds.

What a Trace Actually Contains

A trace is the full record of one agent run. It is not just the answer; it is the sequence of decisions, tool calls, and reasoning steps that produced the answer. OpenAI's trace grading guidance frames it directly: scoring an end-to-end agent trace - the decisions, tool calls, and reasoning steps - is what explains why an agent succeeded or failed, not just whether it did.

That distinction is the whole point. Two runs can produce identical final text. One took a clean path. The other got there by accident after three wrong turns. If you only grade the text, you cannot tell them apart, and the lucky run will eventually fail in production where you are not watching.

So the things worth grading live inside the trace:

• Tool-call correctness - did the agent call the right tool with the right arguments?
• Trace completeness - did it perform every step the task required, or skip one?
• Approval compliance - did it pause for human confirmation on write or destructive actions?
• Retries - did it loop on a failing call, and did it back off sensibly?
• Cost and latency - how many tokens and seconds did the path consume?
• Side-effect safety - did it touch only what it was allowed to touch?

None of these are visible in the final answer. All of them predict whether the agent is safe to scale.

The Anatomy of an Agent Eval

Before writing cases it helps to share vocabulary. Anthropic's guide to evals for AI agents defines a clean set of building blocks - tasks, trials, graders, transcripts, outcomes, eval suites, and harnesses - and the table below maps them to what you actually build.

Two of these deserve emphasis. Run multiple trials per task, because agents are nondeterministic and a single pass tells you little. And choose graders deliberately: a code grader checking "was tool X called with argument Y" is cheap and reliable, while an LLM judge is flexible but needs its own validation. You learn to combine both in AI Evals, Observability & Red-Teaming.

Capability Evals and Regression Evals Are Different Jobs

A common mistake is running one eval suite and expecting it to answer two unrelated questions. Anthropic's guide separates them, and the separation changes how you read the results.

Capability evals measure how good the agent can get. They are allowed to be hard, and you expect partial scores - 60% today, 70% after tuning. They tell you whether a harder task type is within reach.

Regression evals protect what already works. These should sit near a 100% pass rate, because every case in the suite is a behavior you have already shipped and promised. A regression failure is not "the model is still learning." It is "we just broke something users rely on."

Source: Anthropic, "Demystifying evals for AI agents," checked 2026-06-14.

Mixing them produces muddy decisions. If your suite scores 82%, is that a strong capability result or a catastrophic regression result? You cannot tell. Keep two suites, read them against two different bars, and the ship-or-block decision becomes obvious.

The Trace-Based Eval Loop

The mechanics fit a simple loop. The eval suite supplies tasks; the agent runs them; each run emits a trace; graders score the trace, not just the answer; the scores drive a decision.

The arrow that earns its place is AgentRun --> Trace. In a chatbot eval that arrow would point straight at the answer. Here the trace is captured first, and graders read the path. This is also why the harness matters: it has to record tool calls and approvals in a structured form, which the OpenAI Agents SDK is built to emit: agents that plan, call tools, and keep state across steps.

A 12-Case Eval Set for a Knowledge Assistant

Concrete beats abstract. Here is a regression-flavored eval set for an internal knowledge assistant that retrieves company documents, answers questions, and occasionally drafts replies. Each case names what the trace must show, not just what the answer should say.

Notice how many of these cases are invisible to answer-only grading. Case 4 can return a perfectly fluent answer while having leaked a restricted document. Case 8 can return "Done, I updated the record" while having skipped the approval gate. Only the trace reveals the failure.

Consider Quillstone Research, a hypothetical mid-size company whose ops team runs an internal assistant over internal policies and customer accounts. Their answer-only suite had scored 94% for a month, so they shipped a new model. Two days later support flagged that the assistant had quietly answered a question using a contract scoped to a different client — Case 4, leaked verbatim. When they rebuilt the suite around traces, the permission-check assertion failed on the very first trial: the model was retrieving content before the access check, and only the answer text had hidden it. The fix took an afternoon; the trace had made a silent failure loud.

Turning Cases Into Graders

Each case needs a grader, and the cheapest reliable grader is code. "Was the approval step present before the write tool call?" is a boolean you can assert against the trace. "Did the permission check run before retrieval returned content?" is another. Reserve LLM-as-judge for the genuinely fuzzy checks - tone, helpfulness, whether a refusal was phrased well - and validate the judge against human labels before you trust it.

A practical grader checklist for the set above:

• Tool-call assertion: required tool was called with expected arguments
• Order assertion: permission and approval checks ran before the sensitive action
• Refusal assertion: unanswerable and out-of-scope cases did not fabricate
• Citation assertion: every claim maps to a retrieved source that supports it
• Budget assertion: tool-call count and token cost stayed under threshold
• Idempotency assertion: retries did not duplicate a write side effect
• Trial count: each case ran enough trials to expose nondeterminism

This checklist is reusable across products. Swap the document corpus for a CRM, a codebase, or a support inbox, and the same seven assertions still describe a trustworthy run. Building these into a delivery pipeline is the focus of AI Delivery Systems.

Reading the Scores Without Fooling Yourself

Two failure modes sink eval programs. The first is grading optimism: an LLM judge that rubber-stamps runs because it was prompted too generously. Validate judges against a human-labeled sample, and prefer code graders wherever the check is objective.

The second is the flaky-pass trap. Because agents are nondeterministic, a case that passes one trial and fails the next is not really passing. Run multiple trials, report the pass rate per case, and treat anything below your regression bar as a real failure - not noise to be re-run away.

When the suites are clean, the decision is mechanical. Regression suite below ~100%: block and fix. Capability suite climbing toward target: keep tuning. Both green: ship. The loop does its job precisely because the trace, not the answer, decided each outcome.

Where This Goes Next

As agents take on more write actions and longer task chains, the gap between "the answer looked right" and "the run was safe" only widens. Trace-based evals are how that gap stays visible before it reaches a customer. Start small - one regression suite, code graders, a handful of high-stakes cases like injection and write-approval - and grow the set as the agent earns more autonomy.

The day you can point at a failed trace and say exactly which tool call broke the run is the day "the answer looked right" stops being your release criterion. Learn to design and grade those traces in AI Evals, Observability & Red-Teaming, then build agents whose paths are clean enough to grade in the first place with AI Agentic Patterns.

Originally published April 16, 2026. Updated and re-verified June 14, 2026.