On this page
Read
The useful mental model
An eval is a set of tasks with clear success checks. Each task is run more than once because outputs vary, the grading looks at what the agent produced, and the results are kept so you can see whether a change helped. Graders can be code, another model, or a person.
In plain English
It is the exam. You give the agent real tasks, mark the results, and see where it fails, so you know what to fix.
Where builders use it
- 01Checking a prompt or agent change before shipping
- 02Catching regressions after an update
- 03Comparing two versions of an agent
Why it matters
An agent can look good in a demo and still fail in daily use. Evals turn that into numbers you can track, and they stop one improvement from quietly breaking an earlier case.
Why now
Agents run many steps, call tools, and change state, so reading one answer is no longer enough to judge them. Teams building agents are moving from ad hoc checks to repeatable evals. Practices differ between teams, so check current guidance before adopting a specific method.
A common misconception
“Evals are just tests.”
Tests check that software works. Evals check how well an AI system does the job, usually across several runs because the output can differ each time. In AI Garage's framing, tests check the software and evals check whether the agent is getting better.
“One passing run means it works.”
Outputs vary, so teams measure across several trials. They report either the chance of at least one success or the chance that every trial succeeds, depending on whether one success is enough.
See
See the mental model
The visual is generated from the same structured knowledge as the written explanation.
Two columns compare a skill and an eval. A skill teaches the agent how to approach a task and contains instructions, references, and scripts. An eval checks whether the agent can do the task and contains tasks, graders, and scores. Each improves the other.
Builder example
Know whether the agent is getting better
A team's agent drafts support replies. They change its instructions and it seems better.
Failure
They checked three examples by eye. A week later a case they never tried regresses, and nobody notices until a customer does.
Intervention
Collect about twenty to fifty real past failures as tasks, write a clear pass check for each, using code where possible and a rubric or a person where not. Run each task several times and rerun the whole set after every change.
Observable result
Each change shows which tasks improved and which regressed, and the team keeps the changes that raise the score without breaking earlier cases.
Sources
Go deeper
- Anthropic Engineering - Demystifying evals for AI agents
- Anthropic Engineering - Equipping agents for the real world with Agent Skills
Last researched 12 October 2026