Evals for AI Agents: From Vibes to Evidence
Every change to a prompt, a tool or a model changes your agent's behaviour, and « I tried it a few times and it seemed fine » is how regressions reach users. Evals turn that feeling into a number you can trust. You'll build a golden set from real cases, choose graders that check outcomes instead of wording, measure pass rates instead of single runs, and wire it all into CI as a gate. You leave with a working harness you can point at your own agent.
New
Offered by ITwithDom- Last updated November 2026
- Language: English
- 6 h
- E-learning
What you'll learn
- Define an eval as dataset + task + grader + threshold, and know when you need one
- Build a golden set from real logs, past bugs and adversarial cases, with a held-out split
- Choose between code-based, model-based and human graders, and calibrate a model grader
- Evaluate agents by outcomes and trajectories (tool calls, cost), with pass rates over repeated runs
- Gate prompt, tool and model changes in CI with thresholds and a budget
Practical information
- Format
- E-learning
- Location
- Online
- Schedule
- Self-paced: about 1 h 15 of lessons, plus 4–6 h for the exercises and the final project.
Course content
6 lessons, 1 h 15 in total
- Why evals: from vibes to evidence10 min
- A golden set from real cases14 min
- Graders: code first, then models, then people14 min
- Evaluating agents, not just answers13 min
- Evals as a CI gate11 min
- Final project: an eval suite for your agent13 min
Description
A hands-on course on evaluating AI agents: golden sets from real data, code-based and model-based graders, evaluating trajectories and outcomes instead of wording, flakiness and pass rates, and regression gates in CI on every prompt, tool or model change.
Key information
- Learning objectives
- Create a versioned golden set with expected outcomes or rubrics
- Implement at least two grader types and calibrate a model grader against human labels
- Run an agent eval harness with claude -p and structured output
- Add an eval regression gate to CI
- Who it is for
- Developers and ML/AI engineers who ship prompts or agents to real users, and tech leads who decide when a change is good enough.
- Prerequisites
- Basic JavaScript or TypeScript (you write small node:test checks and extend a Node script; the kit is Node-only); macOS, Linux or Windows with WSL 2; git, and Docker for running the harness in a container. Node.js 22.6+, Claude Code 2.1.295+ and an Anthropic API key with a low spend limit (a Console account with credits): the harness exercises (2, 2b, 3, 4) run in a container where you must not log in, so a subscription login alone isn't enough. It is enough for the model grader in exercise 2b on your machine. Each agent run is capped at $1 and cost about $0.25 in our tests: exercise 2 is 15 runs, each CI PR 10 runs, the final project 30 cases × 3 runs (use --k 1 while you iterate).
- Duration
- 6 h
- Price
- Free
- Teaching format
- E-learning
- Access conditions and lead time
- Free. Create a learner account on the course page and start immediately. Immediate access after enrolment.
- Teaching methods
- Short explanations with diagrams, a working eval harness and five graded coding tasks in the kit, four hands-on exercises and a final project on your own agent.
- Assessment
- Short checkpoint quiz on every page (one mistake allowed), final exam at 70 %, Three hands-on exercises with a rubric, Final project: an eval suite with a CI gate, reference solution provided
- Accessibility for people with disabilities
- Self-paced, no time limit. Diagrams have text descriptions, all code is text.
- Contact
- Dominik Heinrich (ITwithDom)
The training provider
ITwithDomSee the profile and all its courses More courses from ITwithDom
See all
Evals for AI Agents: From Vibes to Evidence
Free