Skip to main content
Skip to content
Menu

AI Programming Intermediate 3 parts

Build an eval-driven feedback loop to improve your AI agent's harness

You'll have a documented set of 10-15 production-realistic test cases that expose gaps in your agent's current behavior, plus a prioritized list of harness edits to test against those evals.

What you'll make

Agent Improvement Pipeline

  1. Production Behavior Inventory: 10-15 test cases
  2. Harness Modification Log: changes tested against evals
  3. Performance Evidence: before/after comparisons with generalization checks

The 3 parts

Each part ends with a check you can see.

  1. Mine Your Production Patterns

    Extract test cases from your agent's actual usage to encode what good behavior looks like.

    Done when

    You have 10-15 test cases sourced from real agent interactions, each testing a specific behavioral requirement.

  2. Modify Your Agent Under Test

    Make specific harness changes and test them against your behavioral evals to measure improvement.

    Done when

    You have tested 3-5 harness modifications, documented which changes improved performance on your evals, and can explain why each change worked or failed.

  3. Check for Reward Hacking

    Test whether your agent improvements generalize beyond your specific evals or just game the tests.

    Done when

    You have evidence that your best harness changes improve agent behavior on held-out scenarios, not just your original test cases.

    Capstone: you put together your Agent Improvement Pipeline.

Where this helps

An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.

About this lesson

What you'll be able to do

  • Build 10-15 realistic test cases that expose gaps in your current agent's behavior
  • Engineer specific harness improvements that make your agent pass those evals
  • Run an eval-driven improvement loop that yields measurable performance gains
  • Distinguish agent reward hacking from genuine capability improvements

Who it's for

  • Engineers building AI agents who want systematic improvement methods
  • Developers with agent prototypes ready to scale but unsure how to improve them reliably
  • Teams moving from demo agents to production who need structured testing approaches

What it won't cover

  • Building your first agent from scratch
  • Prompt engineering fundamentals
  • Model training or fine-tuning
  • General software testing theory

Skills you'll practise

  • Production trace analysis
  • Behavioral eval design
  • Harness modification under test
  • Reward hacking detection
  • Iterative system improvement