Evals for the rest of us: test your agent like a checklist

You changed one line in your agent's instructions. It fixed the thing that annoyed you. Did it break anything else? Without evals, you find out when a customer does.

"Eval" sounds academic. It isn't. An eval is a checklist you run against your agent: here are some inputs, here's what a good answer must do, did it do it? Pass or fail. That's the whole idea. This post shows how to build one in an afternoon, run it by hand, then let Claude Code run it for you.

The running example is a support-reply skill: it reads a customer email and drafts a reply. Swap in your own agent; the method is the same.

What an eval is, in plain English

Each test has three parts:

• An input: a real customer email, a real transcript, a real request.
• Criteria: what a good output must do and must not do.
• A grader: whoever or whatever decides pass or fail. That can be you, a small script, or another model call.

Anthropic's guide to agent evals uses the same shape. It calls one test a task, one attempt at it a trial, and the scoring logic a grader. Because AI output varies from run to run, you sometimes run the same task a few times.

Build a small test set from real examples

Don't invent test cases. Collect them. Your agent's history is full of them: the reply that was perfect, the one that promised a refund you don't offer, the one that ignored the customer's question.

Start small. Anthropic's engineering team says 20 to 50 simple tasks drawn from real failures is a great start. If that sounds like a lot, start with 10 and add one every time something goes wrong.

Aim for a mix:

• Normal cases. The everyday requests your agent handles.
• Past failures. Every bug you've fixed becomes a test so it stays fixed.
• Should-not cases. Inputs where the right move is to not do something: decline, ask a question, hand off to a human. Test only one side and you'll get an agent that does the thing all the time.
• Awkward edges. Empty input, a very long email, two questions at once, an angry customer, a message in another language.

Strip personal details before you save anything. Real names, emails and order numbers don't belong in a test folder.

Write pass/fail criteria

Good criteria are specific enough that two people would agree on the verdict. Vague criteria give you vague results.

Some criteria are easy for a script to check (word count, banned phrases). Others need judgment (did it answer the actual question?). That split decides how you grade them.

Run them by hand first

Before you automate anything, do one round yourself. Put the cases in a spreadsheet: input in one column, criteria in the next, then run your agent on each input and mark pass or fail.

This round teaches you more than any script. You'll find criteria that are unclear ("what counts as answering the question?"), cases where two answers are both fine, and failures you didn't know you had. Anthropic's advice here is blunt: read the transcripts. If a test fails every time, suspect the test before you blame the agent.

Fix the criteria until you'd trust someone else to grade with them. Then automate.

Run them with Claude Code

Claude Code has a non-interactive mode: claude -p "your prompt" runs once, prints the answer and exits. That makes it easy to run every case from a script. You can also call a skill this way by putting its command in the prompt.

Lay the tests out as folders:

evals/
├── run.sh
├── history.tsv
└── cases/
    ├── 01-refund-outside-window/
    │   ├── input.md
    │   └── rubric.md
    └── 02-two-questions/
        ├── input.md
        └── rubric.md

rubric.md is your criteria for that case, written as a short list. Then the script:

#!/bin/bash
# Runs every case, grades it, and logs the score.
set -uo pipefail
STAMP=$(date +%Y-%m-%d-%H%M)
OUT="evals/runs/$STAMP"
mkdir -p "$OUT"
pass=0; total=0

SCHEMA='{"type":"object","properties":{"pass":{"type":"boolean"},"reason":{"type":"string"}},"required":["pass","reason"]}'

for dir in evals/cases/*/; do
  name=$(basename "$dir")
  total=$((total + 1))

  # 1. Run the agent on the input
  claude -p "/support-reply ${dir}input.md" --max-turns 5 > "$OUT/$name.md"

  # 2. Script checks: cheap and exact
  words=$(wc -w < "$OUT/$name.md")
  if [ "$words" -gt 150 ]; then
    printf '%s\tFAIL\ttoo long: %s words\n' "$name" "$words" >> "$OUT/results.tsv"; continue
  fi
  if grep -qiE "guarantee|I promise|refund (is|has been) approved" "$OUT/$name.md"; then
    printf '%s\tFAIL\tbanned phrase\n' "$name" >> "$OUT/results.tsv"; continue
  fi

  # 3. Model check against the rubric, with no tools
  verdict=$( { cat "${dir}rubric.md"; echo "--- REPLY ---"; cat "$OUT/$name.md"; } \
    | claude -p "Grade the reply against the rubric. Pass only if every point is met." \
        --tools "" --output-format json --json-schema "$SCHEMA" \
    | jq -r '.structured_output | [(if .pass then "PASS" else "FAIL" end), .reason] | @tsv')

  printf '%s\t%s\n' "$name" "$verdict" >> "$OUT/results.tsv"
  [[ "$verdict" == PASS* ]] && pass=$((pass + 1))
done

commit=$(git rev-parse --short HEAD 2>/dev/null || echo "no-git")
printf '%s\t%s\t%s/%s\n' "$STAMP" "$commit" "$pass" "$total" | tee -a evals/history.tsv

What's happening:

• Step 1 runs your skill on each input. --max-turns 5 stops a run that wanders off.
• Step 2 does the checks a script can do perfectly: word count and banned phrases. They're free and never disagree with themselves.
• Step 3 asks Claude to grade the judgment calls. --tools "" turns off every built-in tool, so the grader only reads what it's given. --json-schema forces the answer into a fixed shape, which lands in the structured_output field. jq (a small JSON tool) pulls it out.

Run it with bash evals/run.sh. Read the failures in evals/runs/ the first few times. If the grader gets it wrong, tighten the rubric, the same way you did by hand. Each run costs real tokens, so start with a handful of cases and grow.

Or check outputs with a hook

A hook is a command Claude Code runs automatically at a set moment (the full tour is in Hooks give agents taste). If your agent writes files, say drafts in a drafts/ folder, a PostToolUse hook can run your script checks every time it writes one:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Write",
        "hooks": [
          {
            "type": "command",
            "command": "\"$CLAUDE_PROJECT_DIR\"/evals/check-draft.sh"
          }
        ]
      }
    ]
  }
}

The hook receives details of the write as JSON on standard input, including tool_input.file_path. Your check-draft.sh reads that path, skips anything outside drafts/, and runs the same word-count and banned-phrase checks. If a check fails, it prints the reason to standard error and exits with code 2, which shows the reason to Claude so it can fix the draft. Evals tell you how the agent is doing overall; a hook catches problems in the moment.

Track changes over time

The last line of run.sh appends a row to evals/history.tsv: when you ran it, which version of your files (the git commit), and the score.

2026-08-20-0915   a1b2c3d   14/18
2026-08-22-1640   d4e5f6a   16/18
2026-08-25-1102   9f8e7d6   12/18

That third row is the whole point. Something you changed made it worse, and you know which commit did it.

A few habits that keep this useful:

• Run it before you ship a change to the prompt, the skill, the model, or the tools.
• Run important cases more than once. Output varies between runs. If something passes 2 times out of 3, it's not reliable yet, and a customer-facing agent needs to be.
• Split "can it?" from "does it still?" Anthropic separates capability evals (hard tasks you're working toward, low scores expected) from regression evals (things that already work, which should stay near 100%). When a hard task starts passing reliably, move it to the regression set.
• Add a case for every new failure. Your test set should grow with your agent.

If you're testing a Claude Code skill specifically, Anthropic's skill-creator plugin automates a fuller version of this loop, including comparing results with and without the skill. Start with the checklist above first. It's small enough that you'll actually run it.

Sources

• Demystifying evals for AI agents (Anthropic Engineering)
• Define success criteria and build evaluations (Claude docs)
• Run Claude Code programmatically (Claude Code docs)
• CLI reference (Claude Code docs)
• Automate actions with hooks (Claude Code docs)
• Extend Claude with skills: evaluate and iterate (Claude Code docs)