Evaluation and Trust

How to tell whether a system that answers, searches or acts is actually working: building question sets with known answers, deciding what counts as good before you look, having a model grade a model, and measuring reliability across many runs rather than trusting one good one.

1. 01Why one good answer proves nothingThe move from judging a single run to judging a system across many: one success is evidence about that run and nothing more.about 7 minutes
2. 02Questions you already know the answers toBuilding a set of real, hard and edge-case questions with known right answers, and the trap of tuning a system until it merely passes its own test.about 8 minutes
3. 03Deciding what "good" means before you checkPass/fail against graded criteria, what failure rate is acceptable, cost and speed alongside correctness, and deciding before you look rather than after.about 8 minutes
4. 04Using a model to grade a modelA second model reading output against a written rubric, sending failures back for revision, and checking the grader itself against a person's judgement.about 8 minutes
5. 05Measuring a retrieval systemHow often the right passage comes back across many questions, and whether answers stay grounded in what was retrieved rather than in what the model already held.about 7 minutes
6. 06Measuring an agent: the path, not just the destinationJudging whether an agent reliably takes a sound route across many runs, not only whether it lands on an acceptable answer; a second section on tool-connected systems held for Module 4.about 8 minutes
7. 07Guardrails: limits you check, not limits you hope forChecks on what goes into a system and what comes out of it, the boundaries it is meant to refuse to cross, and testing where each of those checks actually fails.about 7 minutes
8. 08Bias, change, and when to trust a systemBias in output across many uses, re-checking a system after what it runs on changes, and why trust is maintained through ongoing checks rather than granted once.about 7 minutes

Opening a lesson needs membership. During the preview it's free for invited members. See plans