# Evals

> Part of the AI Garage Jargons glossary: https://aigarage.in/jargons (all terms: https://aigarage.in/jargons.md)

**Category:** Machine Learning
**Tags:** evaluation, agents, testing, quality

Tests for an AI system: you give it an input, then apply grading logic to its output to measure how well it did the job.

**In plain terms:** It is the exam. You give the agent real tasks, mark the results, and see where it fails, so you know what to fix.

**Use cases:**
- Checking a prompt or agent change before shipping
- Catching regressions after an update
- Comparing two versions of an agent

**Why it matters:** An agent can look good in a demo and still fail in daily use. Evals turn that into numbers you can track, and they stop one improvement from quietly breaking an earlier case.

**Why now:** Agents run many steps, call tools, and change state, so reading one answer is no longer enough to judge them. Teams building agents are moving from ad hoc checks to repeatable evals. Practices differ between teams, so check current guidance before adopting a specific method.

**AI Garage example:** The AI Garage Jargons workflow already uses eval-like checks. A script audit scores each narration script against fixed targets such as length, reading ease, and closing length, and a video review scores each render against a rubric with a pass bar. This describes checks used in this repository, not a claim about production usage.

**Mental model:** An eval is a set of tasks with clear success checks. Each task is run more than once because outputs vary, the grading looks at what the agent produced, and the results are kept so you can see whether a change helped. Graders can be code, another model, or a person.

**Builder scenario: Know whether the agent is getting better**

- **Context:** A team's agent drafts support replies. They change its instructions and it seems better.
- **Failure:** They checked three examples by eye. A week later a case they never tried regresses, and nobody notices until a customer does.
- **Intervention:** Collect about twenty to fifty real past failures as tasks, write a clear pass check for each, using code where possible and a rubric or a person where not. Run each task several times and rerun the whole set after every change.
- **Outcome:** Each change shows which tasks improved and which regressed, and the team keeps the changes that raise the score without breaking earlier cases.

**Common misconceptions:**
- **Claim:** Evals are just tests.
  **Correction:** Tests check that software works. Evals check how well an AI system does the job, usually across several runs because the output can differ each time. In AI Garage's framing, tests check the software and evals check whether the agent is getting better.
- **Claim:** One passing run means it works.
  **Correction:** Outputs vary, so teams measure across several trials. They report either the chance of at least one success or the chance that every trial succeeds, depending on whether one success is enough.

**Visual model (comparison):** The eval is the exam that shows what the skill still gets wrong. Fix the recurring failures in the skill, then run the eval again. This is AI Garage's framing of the relationship.

**Visual description:** Two columns compare a skill and an eval. A skill teaches the agent how to approach a task and contains instructions, references, and scripts. An eval checks whether the agent can do the task and contains tasks, graders, and scores. Each improves the other.

**Related concepts:** [Prompt Engineering](https://aigarage.in/jargons/prompt-engineering), [Hallucination (AI)](https://aigarage.in/jargons/ai-hallucination), [Fine-tuning](https://aigarage.in/jargons/fine-tuning), [Agent Skill](https://aigarage.in/jargons/agent-skill), [AI Agent](https://aigarage.in/jargons/ai-agent)

**Last researched:** 2026-10-12

**Sources:**
- [Anthropic Engineering - Demystifying evals for AI agents](https://anthropic.com/engineering/demystifying-evals-for-ai-agents)
- [Anthropic Engineering - Equipping agents for the real world with Agent Skills](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills)

**URL:** https://aigarage.in/jargons/evals