
Audit your AI agent's decision quality by measuring exploration gaps
You'll have a diagnostic framework that identifies whether your agent is failing because it didn't gather enough information (exploration) or because it acted poorly on available info (exploitation),
What you'll make
Agent Decision Quality Audit
- Decision Log Analysis: patterns in available vs missing information
- Exploration Gap Measurements: specific information deficits
- Failure Classification: exploration vs exploitation errors
- Architecture Recommendations: changes to fix identified patterns
Submit your analysis as structured findings with evidence from your agent's actual decision logs
The 4 parts
Each part ends with a check you can see.
Measure Your Agent's Information Access
Run specific tests to quantify what information your agent actually gathers before making decisions.
Done when
You have 3+ quantified measurements showing when your agent makes decisions without accessing available information sources, with at least one measurement showing a significant gap.
Classify Recent Failure Patterns
Analyze your agent's recent failures to determine whether they stem from missing information or poor use of available information.
Done when
You have classified 3-5 recent agent failures as exploration (missing info) or exploitation (bad decisions on good info) with evidence for each classification.
Test Information-Gathering Fixes
Implement and measure small changes to your agent's information-gathering behavior based on your gap measurements.
Done when
You have tested at least 2 specific changes to your agent's exploration behavior and measured whether they improve decision quality without breaking other functions.
Build Your Diagnostic Framework
Create a reusable process for diagnosing whether future agent problems are exploration or exploitation errors.
Done when
You have a diagnostic framework with specific steps, measurements, and decision criteria for classifying agent failures going forward.
Capstone: you put together your Agent Decision Quality Audit.
Where this helps
An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.
About this lesson
What you'll be able to do
- Distinguish exploration from exploitation errors in your AI agent's failures
- Measure information-gathering gaps using decision logs
- Design diagnostic tests that reveal why your agent fails
- Make and defend architecture changes based on error patterns
Who it's for
- Developers running AI agents that make inconsistent decisions
- Product teams whose agents work sometimes but fail mysteriously
- Anyone debugging agent behavior beyond prompt tweaks
What it won't cover
- How to build agents from scratch
- Prompt engineering fundamentals
- Agent security or deployment infrastructure
- General LLM evaluation methods
Skills you'll practise
- Decision log analysis
- Exploration gap measurement
- Failure pattern classification
- Agent diagnostic design