Measuring whether an agent is working
How to build a small test set, pick graders, track the right numbers in production, and tell a lucky agent from a reliable one.
Updated 2026-11-19. "(source N)" refers to the numbered list at the end.
Why gut feel is not enough
Most agent projects are judged by someone trying it a few times and saying it seems fine. Then a new model comes out, or someone edits the prompt, and nobody can say whether things got better or worse.
Quality is the biggest reason agents fail to ship. In LangChain's practitioner survey, about a third of respondents named it as their main barrier, more than speed or cost (source 2). The fix is a habit, not a product: write down what good looks like, test against it, and keep testing.
Step 1: define success in numbers
Anthropic's guidance says good success criteria are specific, measurable, achievable and relevant (source 5). Accurate replies is too vague. Ninety per cent of refund requests get the correct decision, and none promise a refund the policy does not allow, is testable.
Most agents need several measures at once (source 5). A support agent might be judged on:
- Task success: did it reach the right outcome?
- Safety: did it avoid actions it should never take?
- Escalation: did it hand off the cases it should?
- Speed: how long did the customer wait?
- Cost: how much did each ticket cost in model fees?
Step 2: build a small test set
You do not need hundreds of tests to start. Anthropic's engineering team says 20 to 50 small tasks taken from things that actually broke is a good start (source 1). Pull them from support tickets, bug reports and the checks you already do by hand.
Make each task clear enough that two experts would agree on pass or fail, and write a reference answer for each (source 1). Include cases where the agent should act and cases where it should refuse or escalate, so you do not train it to always say yes.
OpenAI gives the same advice for its tools: use test data that looks like real traffic, include human-checked answers, start small and grow (source 4).
Step 3: choose your graders
A grader is whatever scores a result. Anthropic describes three kinds (source 1).
- Code checks: exact matches, unit tests, or checking the final state, such as whether the refund record exists. Fast, cheap and repeatable, but rigid.
- Model graders: another model scores the output against a written rubric. Flexible, but they cost money and need checking against human judgment.
- Human review: the gold standard, but slow and costly. Use it to spot-check and to calibrate the model graders.
Which grader when
The advice is to use code checks where you can, model graders where you must, and people sparingly (source 1). LangChain's survey shows teams already mix them: 59.8% use human review and 53.3% use model graders (source 2).
Grade the outcome, not the exact path. If the agent found a different but valid way to solve the task, it should pass (source 1).
Step 4: test for consistency
Agents give different answers on different runs. One success proves little. Run each task several times.
Two measures help (source 1). Pass@k is the chance of at least one success in k tries, which suits work where a person picks the best of several drafts. Pass^k is the chance that all k tries succeed, which is what matters for anything customer-facing. An agent that succeeds 75% of the time passes three runs in a row only about 42% of the time.
The pass^k idea comes from the tau-bench research benchmark. Its authors found even strong agents of the time were inconsistent, with pass^8 below 25% on a retail task set (source 3).
Step 5: keep two kinds of suite
Anthropic separates capability tests from regression tests (source 1). Capability tests are hard tasks the agent cannot do yet; they should start with a low pass rate. Regression tests are things it already does well; they should stay near 100%. When a capability test passes reliably, move it into the regression suite.
Run the regression suite before every prompt change, model swap or tool change.
Step 6: read the transcripts
Scores lie when graders break. Anthropic warns that a 0% pass rate across many runs usually means a broken task, not a hopeless agent, and that no score should be trusted until someone has read the details (source 1). Read a handful of transcripts every week.
In production: watch the live numbers
Testing before launch is half the job. LangChain found 52.4% of teams run offline tests on fixed data and 37.3% score live traffic too (source 2). Among teams with agents in production, 94% have some monitoring and 71.5% have full tracing of each step (source 2).
Useful live numbers for a small business include: share of tasks completed without a person, share escalated, share a person had to correct, average cost per task, and complaints per hundred tasks.
Keep your tests portable
Store test cases in your own files, not only inside a vendor's dashboard. OpenAI's hosted Evals platform, for example, goes read-only on 31 October 2026 and shuts down on 30 November 2026 (source 4). Tools change; your test set is the asset that lasts.
Sources
Written by Agents Lookup in our own words. Facts come from these pages:
- 1. Anthropic Engineering: Demystifying evals for AI agents: www.anthropic.com/engineering/demystifying-evals-for-ai-agents
- 2. LangChain: State of Agent Engineering: www.langchain.com/state-of-agent-engineering
- 3. arXiv: tau-bench, a benchmark for tool-agent-user interaction in real-world domains: arxiv.org/abs/2406.12045
- 4. OpenAI: Evals guide: developers.openai.com/api/docs/guides/evals
- 5. Anthropic: Define your success criteria: platform.claude.com/docs/en/test-and-evaluate/define-success