Skip to main content
Skip to content
Menu

Agents Intermediate 4 parts

Build feedback loops that systematically improve agent performance over time

You'll have a repeating process where you evaluate agent outputs, feed results back into training signals, and measure performance improvements across successive versions.

What you'll make

Agent Improvement Loop

  1. Trace Capture Implementation: logging decision points and context
  2. Feedback Integration: automated evals and failure pattern detection
  3. Evaluation Dataset: offline tests from real agent behavior
  4. CI/CD Validation: automated testing pipeline for agent changes

Submit your Python implementation with working trace capture, one feedback mechanism, and test results showing measurable improvement

The 4 parts

Each part ends with a check you can see.

  1. Capture Agent Decision Traces

    Build a system that records what your agent considers at each decision point, not just what it outputs

    Done when

    You have trace data showing agent reasoning steps, tool calls, and context for 10+ real interactions with measurable decision points

  2. Identify Recurring Failure Patterns

    Analyze your trace data to find systematic problems that repeat across multiple agent interactions

    Done when

    You have identified 2-3 specific failure patterns with evidence from traces, plus automated detection for at least one pattern

  3. Build Offline Evaluation Dataset

    Convert your agent's real-world behavior into test cases that measure improvement without running live experiments

    Done when

    You have 15+ test cases from real agent interactions with expected outputs, plus measurable scoring that shows before/after performance differences

  4. Automate Improvement Validation

    Build a CI/CD process that tests agent changes against your evaluation dataset before production deployment

    Done when

    You have working automation that runs your evaluation dataset on agent changes and blocks deployment if performance drops below baseline

    Capstone: you put together your Agent Improvement Loop.

Where this helps

An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.

About this lesson

What you'll be able to do

  • Build a trace capture system that records agent decision points and outcomes
  • Create feedback loops that identify and fix recurring agent failure patterns
  • Validate agent improvements using offline evaluation datasets
  • Establish CI/CD processes for testing agent changes before production

Who it's for

  • Engineers building agent systems who need systematic improvement processes
  • Teams shipping agents who want data-driven performance optimization
  • Technical leads responsible for agent reliability in production

What it won't cover

  • Agent architecture fundamentals
  • Multi-agent coordination patterns
  • Payment integration for autonomous agents
  • Advanced prompt engineering techniques

Skills you'll practise

  • Trace data analysis
  • Feedback loop design
  • Performance measurement
  • Systematic evaluation
  • Process automation