
Build an automated pipeline that re-evaluates your current model against new releases monthly
You'll have a scheduled script (cron/GitHub Actions/cloud function) that runs your team's standard evaluation suite on 2-3 new frontier models, logs results, and flags when performance improves enough
What you'll make
Model Upgrade Pipeline
- evaluate_models() — runs your eval suite on multiple models: what to include
- log_results() — structured comparison output: what to include
- flag_upgrades() — decision logic for when to alert: what to include
- schedule_run() — automation wrapper: what to include
Paste your working Python script with at least one scheduled run completed and results logged
The 4 parts
Each part ends with a check you can see.
Baseline Your Current Model
Run your existing evaluation suite on your current model and document the performance profile that matters for your deployment.
Done when
A documented baseline with specific metrics, runtime constraints, and the 2-3 performance dimensions you actually care about for your use case
Test New Releases Against Baseline
Run the same evaluation suite on 2-3 recent model releases and compare results systematically against your baseline.
Done when
Side-by-side comparison of at least 2 new models vs your baseline with documented wins/losses and deployment feasibility for each
Codify Upgrade Decision Rules
Convert your manual upgrade decisions into explicit rules that can flag when a new model is worth switching to.
Done when
Working decision logic that takes evaluation results and outputs upgrade recommendations with reasoning, tested on your comparative data
Schedule and Deploy Pipeline
Wire together evaluation, comparison, and decision logic into a scheduled pipeline that runs monthly and logs results.
Done when
Complete pipeline that runs automatically, tests new models, logs structured results, and flags upgrade opportunities with at least one successful execution
Capstone: you put together your Model Upgrade Pipeline.
Where this helps
An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.
About this lesson
What you'll be able to do
- Run a working evaluation pipeline that tests your current model against 2-3 new releases automatically
- Identify which performance improvements matter for your actual use case vs which are benchmark noise
- Have a decision framework for when to switch models based on real deployment constraints
- Maintain a log of model performance over time that informs your next upgrade cycle
Who it's for
- Teams already using a specific model in production who want to stay current without manual testing overhead
- AI practitioners comfortable with basic scripting who need systematic model comparison
- Anyone who's wondered 'should we upgrade?' but lacks data to decide
What it won't cover
- Setting up your initial model deployment or evaluation metrics
- Fine-tuning or training custom models
- Comprehensive benchmark recreation - this uses your existing eval suite
- Model architecture analysis or theoretical performance prediction
Skills you'll practise
- Automated evaluation pipeline design
- Model performance comparison across versions
- Deployment constraint mapping
- Performance threshold calibration