
Build an eval-driven feedback loop to improve your AI agent's harness
You'll have a documented set of 10-15 production-realistic test cases that expose gaps in your agent's current behavior, plus a prioritized list of harness edits to test against those evals.
What you'll make
Agent Improvement Pipeline
- Production Behavior Inventory: 10-15 test cases
- Harness Modification Log: changes tested against evals
- Performance Evidence: before/after comparisons with generalization checks
The 3 parts
Each part ends with a check you can see.
Mine Your Production Patterns
Extract test cases from your agent's actual usage to encode what good behavior looks like.
Done when
You have 10-15 test cases sourced from real agent interactions, each testing a specific behavioral requirement.
Modify Your Agent Under Test
Make specific harness changes and test them against your behavioral evals to measure improvement.
Done when
You have tested 3-5 harness modifications, documented which changes improved performance on your evals, and can explain why each change worked or failed.
Check for Reward Hacking
Test whether your agent improvements generalize beyond your specific evals or just game the tests.
Done when
You have evidence that your best harness changes improve agent behavior on held-out scenarios, not just your original test cases.
Capstone: you put together your Agent Improvement Pipeline.
Where this helps
An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.
About this lesson
What you'll be able to do
- Build 10-15 realistic test cases that expose gaps in your current agent's behavior
- Engineer specific harness improvements that make your agent pass those evals
- Run an eval-driven improvement loop that yields measurable performance gains
- Distinguish agent reward hacking from genuine capability improvements
Who it's for
- Engineers building AI agents who want systematic improvement methods
- Developers with agent prototypes ready to scale but unsure how to improve them reliably
- Teams moving from demo agents to production who need structured testing approaches
What it won't cover
- Building your first agent from scratch
- Prompt engineering fundamentals
- Model training or fine-tuning
- General software testing theory
Skills you'll practise
- Production trace analysis
- Behavioral eval design
- Harness modification under test
- Reward hacking detection
- Iterative system improvement