Skip to main content
Skip to content
Menu

Codex Intermediate 3 parts Updated

Build a coding benchmark test suite

You'll have a working script that runs your own codebase against AI models and measures which one solves your real tasks fastest.

What you'll make

AI Coding Benchmark Suite

  1. setup_models() — API connections and model configs: what to include
  2. load_test_tasks() — your real coding challenges: what to include
  3. run_benchmark() — execution and timing logic: what to include
  4. analyze_results() — comparison scoring and reporting: what to include
  5. main() — orchestration and output formatting: what to include

Paste your complete Python script that can be run with 'python benchmark.py' to test AI models on your tasks

The 3 parts

Each part ends with a check you can see.

  1. Extract Your Real Coding Tasks

    Pull 8-10 actual coding challenges from your recent work and classify them by structure type

    Done when

    You have 8-10 coding tasks from your real work, each with a clear problem statement and success criteria, sorted into well-specified vs open-ended categories

  2. Build the Execution Harness

    Write the benchmark runner that feeds tasks to AI models, captures responses, and measures timing

    Done when

    You have a working script that can send the same coding task to multiple AI models, capture their responses, time the interactions, and save results to compare

  3. Run Full Benchmark and Analyze Patterns

    Execute your task suite across all models, then analyze which model wins on which task types and why

    Done when

    You have benchmark results from all your tasks across all models, with documented patterns about which model performs better on well-specified vs open-ended tasks in your domain

    Capstone: you put together your AI Coding Benchmark Suite.

Where this helps

An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.

About this lesson

What you'll be able to do

  • Run a working benchmark script on at least 3 AI models with your own real coding tasks
  • Identify which model wins on which type of task in your specific context
  • Have a documented process for re-running benchmarks when new models release

Who it's for

  • Developers who write code for work and want to pick the right AI tool for different tasks
  • Anyone who's tried multiple coding AIs but can't tell which actually helps more

What it won't cover

  • Setting up AI model APIs from scratch (assume you have access)
  • Comprehensive evaluation across all possible programming languages
  • Academic benchmarking methodology — this is practical comparison for your workflow

Skills you'll practise

  • Task decomposition for coding benchmarks
  • Performance measurement across AI models
  • Failure pattern recognition
  • Tool selection criteria development