Skip to main content
Skip to content
Menu

Developer Tools Intermediate 3 parts

Transform your recurring agent failures into automated evaluation datasets

You'll have a reusable dataset of 5–10 real failures your agent encounters, tagged with what went wrong, that you can use to evaluate prompt changes before shipping them to production.

What you'll make

Agent Failure Evaluation Dataset

  1. Failure cases with context: input, expected behavior, actual failure
  2. Categorization schema: failure types and tags
  3. Test runner: code to validate fixes against all cases

Upload your Python test file with documented failure cases and a script that can validate prompt changes

The 3 parts

Each part ends with a check you can see.

  1. Capture Your Failure Patterns

    Extract and categorize real failure cases from your agent's production behavior

    Done when

    You have 5-10 documented failure cases with full context (input, expected vs actual output, failure type)

  2. Design Testable Cases

    Convert failure patterns into automated test cases that can validate prompt improvements

    Done when

    You have test cases for each failure type that specify input, expected behavior, and pass/fail criteria

  3. Test Before You Ship

    Run your evaluation dataset against prompt changes to catch regressions before production

    Done when

    You can test any prompt change against your failure dataset and get a clear pass/fail result in under 5 minutes

    Capstone: you put together your Agent Failure Evaluation Dataset.

Where this helps

An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.

About this lesson

What you'll be able to do

  • Capture and categorize real agent failures from production logs
  • Build testable datasets from recurring failure patterns
  • Validate prompt changes against real failure cases before deploying

Who it's for

  • Developers running AI agents in production who have recurring failures
  • Teams shipping prompt updates without systematic testing
  • Anyone debugging agent behavior manually instead of systematically

What it won't cover

  • How to build agents from scratch
  • LangChain framework specifics
  • General prompt engineering techniques
  • Setting up production monitoring infrastructure

Skills you'll practise

  • Failure pattern recognition
  • Test case generation from real data
  • Automated evaluation design
  • Production validation workflows