Skip to main content
Skip to content
Menu

Agents Intermediate 4 parts

Audit your AI agent's decision quality by measuring exploration gaps

You'll have a diagnostic framework that identifies whether your agent is failing because it didn't gather enough information (exploration) or because it acted poorly on available info (exploitation),

What you'll make

Agent Decision Quality Audit

  1. Decision Log Analysis: patterns in available vs missing information
  2. Exploration Gap Measurements: specific information deficits
  3. Failure Classification: exploration vs exploitation errors
  4. Architecture Recommendations: changes to fix identified patterns

Submit your analysis as structured findings with evidence from your agent's actual decision logs

The 4 parts

Each part ends with a check you can see.

  1. Measure Your Agent's Information Access

    Run specific tests to quantify what information your agent actually gathers before making decisions.

    Done when

    You have 3+ quantified measurements showing when your agent makes decisions without accessing available information sources, with at least one measurement showing a significant gap.

  2. Classify Recent Failure Patterns

    Analyze your agent's recent failures to determine whether they stem from missing information or poor use of available information.

    Done when

    You have classified 3-5 recent agent failures as exploration (missing info) or exploitation (bad decisions on good info) with evidence for each classification.

  3. Test Information-Gathering Fixes

    Implement and measure small changes to your agent's information-gathering behavior based on your gap measurements.

    Done when

    You have tested at least 2 specific changes to your agent's exploration behavior and measured whether they improve decision quality without breaking other functions.

  4. Build Your Diagnostic Framework

    Create a reusable process for diagnosing whether future agent problems are exploration or exploitation errors.

    Done when

    You have a diagnostic framework with specific steps, measurements, and decision criteria for classifying agent failures going forward.

    Capstone: you put together your Agent Decision Quality Audit.

Where this helps

An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.

About this lesson

What you'll be able to do

  • Distinguish exploration from exploitation errors in your AI agent's failures
  • Measure information-gathering gaps using decision logs
  • Design diagnostic tests that reveal why your agent fails
  • Make and defend architecture changes based on error patterns

Who it's for

  • Developers running AI agents that make inconsistent decisions
  • Product teams whose agents work sometimes but fail mysteriously
  • Anyone debugging agent behavior beyond prompt tweaks

What it won't cover

  • How to build agents from scratch
  • Prompt engineering fundamentals
  • Agent security or deployment infrastructure
  • General LLM evaluation methods

Skills you'll practise

  • Decision log analysis
  • Exploration gap measurement
  • Failure pattern classification
  • Agent diagnostic design