
Build a cost-tracking script that compares batch size tradeoffs for your inference workload
You'll have a working Python script that calculates token cost and latency across 3-5 batch size configurations for your actual model deployment, with CSV output showing the economics of each option.
What you'll make
Batch Economics Script
- calculate_batch_costs() — token pricing and throughput calculation: what to include
- measure_latency() — timing across batch sizes: what to include
- generate_report() — CSV output with cost-latency frontier: what to include
- main() — configuration and execution: what to include
Paste your complete Python script that runs end-to-end on your inference workload
The 3 parts
Each part ends with a check you can see.
Measure Your Current Costs
Instrument your actual inference workload to capture token counts, costs, and latency for one week
Done when
You have 7 days of real usage data showing tokens processed, actual costs paid, and response times with timestamps
Test Batch Size Variations
Run the same workload across 3-5 different batch sizes and measure the cost and latency differences
Done when
You have measured data for at least 3 batch sizes showing cost-per-token and average latency for each configuration
Build the Economics Script
Code the cost-tracking script that calculates batch economics and outputs CSV data for decision-making
Done when
A working Python script that takes your measurement data and outputs cost-latency analysis with clear recommendations
Capstone: you put together your Batch Economics Script.
Where this helps
An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.
About this lesson
What you'll be able to do
- Run a working cost-tracking script on your actual inference workload across 3-5 batch size configurations
- Identify the specific batch size where your cost-per-token vs latency tradeoff breaks
- Have documented economics data showing which configuration saves money vs which saves time for your use case
Who it's for
- Anyone running inference workloads who's wondered if they're batching optimally
- People deploying models in production who need to justify infrastructure spending
- Developers who can read Python and want to understand the math behind API pricing
What it won't cover
- Training batch size optimization (this is inference-only)
- Multi-GPU or distributed inference setup
- Model architecture choices or quantization techniques
Skills you'll practise
- Batch economics modeling from real usage
- Cost-per-token calculation across configurations
- Latency vs throughput tradeoff analysis
- Infrastructure cost debugging from API data