Skip to main content
Skip to content
Menu

AI pretraining Advanced 4 parts

Build an automated pipeline that re-evaluates your current model against new releases monthly

You'll have a scheduled script (cron/GitHub Actions/cloud function) that runs your team's standard evaluation suite on 2-3 new frontier models, logs results, and flags when performance improves enough

What you'll make

Model Upgrade Pipeline

  1. evaluate_models() — runs your eval suite on multiple models: what to include
  2. log_results() — structured comparison output: what to include
  3. flag_upgrades() — decision logic for when to alert: what to include
  4. schedule_run() — automation wrapper: what to include

Paste your working Python script with at least one scheduled run completed and results logged

The 4 parts

Each part ends with a check you can see.

  1. Baseline Your Current Model

    Run your existing evaluation suite on your current model and document the performance profile that matters for your deployment.

    Done when

    A documented baseline with specific metrics, runtime constraints, and the 2-3 performance dimensions you actually care about for your use case

  2. Test New Releases Against Baseline

    Run the same evaluation suite on 2-3 recent model releases and compare results systematically against your baseline.

    Done when

    Side-by-side comparison of at least 2 new models vs your baseline with documented wins/losses and deployment feasibility for each

  3. Codify Upgrade Decision Rules

    Convert your manual upgrade decisions into explicit rules that can flag when a new model is worth switching to.

    Done when

    Working decision logic that takes evaluation results and outputs upgrade recommendations with reasoning, tested on your comparative data

  4. Schedule and Deploy Pipeline

    Wire together evaluation, comparison, and decision logic into a scheduled pipeline that runs monthly and logs results.

    Done when

    Complete pipeline that runs automatically, tests new models, logs structured results, and flags upgrade opportunities with at least one successful execution

    Capstone: you put together your Model Upgrade Pipeline.

Where this helps

An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.

About this lesson

What you'll be able to do

  • Run a working evaluation pipeline that tests your current model against 2-3 new releases automatically
  • Identify which performance improvements matter for your actual use case vs which are benchmark noise
  • Have a decision framework for when to switch models based on real deployment constraints
  • Maintain a log of model performance over time that informs your next upgrade cycle

Who it's for

  • Teams already using a specific model in production who want to stay current without manual testing overhead
  • AI practitioners comfortable with basic scripting who need systematic model comparison
  • Anyone who's wondered 'should we upgrade?' but lacks data to decide

What it won't cover

  • Setting up your initial model deployment or evaluation metrics
  • Fine-tuning or training custom models
  • Comprehensive benchmark recreation - this uses your existing eval suite
  • Model architecture analysis or theoretical performance prediction

Skills you'll practise

  • Automated evaluation pipeline design
  • Model performance comparison across versions
  • Deployment constraint mapping
  • Performance threshold calibration