Skip to main content
Skip to content
Menu

AI pretraining Intermediate 3 parts

Build a cost-tracking script that compares batch size tradeoffs for your inference workload

You'll have a working Python script that calculates token cost and latency across 3-5 batch size configurations for your actual model deployment, with CSV output showing the economics of each option.

What you'll make

Batch Economics Script

  1. calculate_batch_costs() — token pricing and throughput calculation: what to include
  2. measure_latency() — timing across batch sizes: what to include
  3. generate_report() — CSV output with cost-latency frontier: what to include
  4. main() — configuration and execution: what to include

Paste your complete Python script that runs end-to-end on your inference workload

The 3 parts

Each part ends with a check you can see.

  1. Measure Your Current Costs

    Instrument your actual inference workload to capture token counts, costs, and latency for one week

    Done when

    You have 7 days of real usage data showing tokens processed, actual costs paid, and response times with timestamps

  2. Test Batch Size Variations

    Run the same workload across 3-5 different batch sizes and measure the cost and latency differences

    Done when

    You have measured data for at least 3 batch sizes showing cost-per-token and average latency for each configuration

  3. Build the Economics Script

    Code the cost-tracking script that calculates batch economics and outputs CSV data for decision-making

    Done when

    A working Python script that takes your measurement data and outputs cost-latency analysis with clear recommendations

    Capstone: you put together your Batch Economics Script.

Where this helps

An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.

About this lesson

What you'll be able to do

  • Run a working cost-tracking script on your actual inference workload across 3-5 batch size configurations
  • Identify the specific batch size where your cost-per-token vs latency tradeoff breaks
  • Have documented economics data showing which configuration saves money vs which saves time for your use case

Who it's for

  • Anyone running inference workloads who's wondered if they're batching optimally
  • People deploying models in production who need to justify infrastructure spending
  • Developers who can read Python and want to understand the math behind API pricing

What it won't cover

  • Training batch size optimization (this is inference-only)
  • Multi-GPU or distributed inference setup
  • Model architecture choices or quantization techniques

Skills you'll practise

  • Batch economics modeling from real usage
  • Cost-per-token calculation across configurations
  • Latency vs throughput tradeoff analysis
  • Infrastructure cost debugging from API data