
Build feedback loops that systematically improve agent performance over time
You'll have a repeating process where you evaluate agent outputs, feed results back into training signals, and measure performance improvements across successive versions.
What you'll make
Agent Improvement Loop
- Trace Capture Implementation: logging decision points and context
- Feedback Integration: automated evals and failure pattern detection
- Evaluation Dataset: offline tests from real agent behavior
- CI/CD Validation: automated testing pipeline for agent changes
Submit your Python implementation with working trace capture, one feedback mechanism, and test results showing measurable improvement
The 4 parts
Each part ends with a check you can see.
Capture Agent Decision Traces
Build a system that records what your agent considers at each decision point, not just what it outputs
Done when
You have trace data showing agent reasoning steps, tool calls, and context for 10+ real interactions with measurable decision points
Identify Recurring Failure Patterns
Analyze your trace data to find systematic problems that repeat across multiple agent interactions
Done when
You have identified 2-3 specific failure patterns with evidence from traces, plus automated detection for at least one pattern
Build Offline Evaluation Dataset
Convert your agent's real-world behavior into test cases that measure improvement without running live experiments
Done when
You have 15+ test cases from real agent interactions with expected outputs, plus measurable scoring that shows before/after performance differences
Automate Improvement Validation
Build a CI/CD process that tests agent changes against your evaluation dataset before production deployment
Done when
You have working automation that runs your evaluation dataset on agent changes and blocks deployment if performance drops below baseline
Capstone: you put together your Agent Improvement Loop.
Where this helps
An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.
About this lesson
What you'll be able to do
- Build a trace capture system that records agent decision points and outcomes
- Create feedback loops that identify and fix recurring agent failure patterns
- Validate agent improvements using offline evaluation datasets
- Establish CI/CD processes for testing agent changes before production
Who it's for
- Engineers building agent systems who need systematic improvement processes
- Teams shipping agents who want data-driven performance optimization
- Technical leads responsible for agent reliability in production
What it won't cover
- Agent architecture fundamentals
- Multi-agent coordination patterns
- Payment integration for autonomous agents
- Advanced prompt engineering techniques
Skills you'll practise
- Trace data analysis
- Feedback loop design
- Performance measurement
- Systematic evaluation
- Process automation