
Build a coding benchmark test suite
You'll have a working script that runs your own codebase against AI models and measures which one solves your real tasks fastest.
What you'll make
AI Coding Benchmark Suite
- setup_models() — API connections and model configs: what to include
- load_test_tasks() — your real coding challenges: what to include
- run_benchmark() — execution and timing logic: what to include
- analyze_results() — comparison scoring and reporting: what to include
- main() — orchestration and output formatting: what to include
Paste your complete Python script that can be run with 'python benchmark.py' to test AI models on your tasks
The 3 parts
Each part ends with a check you can see.
Extract Your Real Coding Tasks
Pull 8-10 actual coding challenges from your recent work and classify them by structure type
Done when
You have 8-10 coding tasks from your real work, each with a clear problem statement and success criteria, sorted into well-specified vs open-ended categories
Build the Execution Harness
Write the benchmark runner that feeds tasks to AI models, captures responses, and measures timing
Done when
You have a working script that can send the same coding task to multiple AI models, capture their responses, time the interactions, and save results to compare
Run Full Benchmark and Analyze Patterns
Execute your task suite across all models, then analyze which model wins on which task types and why
Done when
You have benchmark results from all your tasks across all models, with documented patterns about which model performs better on well-specified vs open-ended tasks in your domain
Capstone: you put together your AI Coding Benchmark Suite.
Where this helps
An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.
About this lesson
What you'll be able to do
- Run a working benchmark script on at least 3 AI models with your own real coding tasks
- Identify which model wins on which type of task in your specific context
- Have a documented process for re-running benchmarks when new models release
Who it's for
- Developers who write code for work and want to pick the right AI tool for different tasks
- Anyone who's tried multiple coding AIs but can't tell which actually helps more
What it won't cover
- Setting up AI model APIs from scratch (assume you have access)
- Comprehensive evaluation across all possible programming languages
- Academic benchmarking methodology — this is practical comparison for your workflow
Skills you'll practise
- Task decomposition for coding benchmarks
- Performance measurement across AI models
- Failure pattern recognition
- Tool selection criteria development