Skip to main content
Skip to content
Menu

Open Source Beginner 3 parts

Evaluate open-source reasoning models on your actual business problems

You'll have a comparison document showing how frontier-class open models perform on 2–3 problems you actually need solved, with cost and latency metrics.

What you'll make

Open Model Evaluation Report

  1. Problem Test Suite: your actual tasks formatted for testing
  2. Model Performance Matrix: accuracy and quality scores per problem
  3. Cost Analysis: per-task costs vs current solution
  4. Deployment Decision: go/no-go with evidence

The 3 parts

Each part ends with a check you can see.

  1. Define Your Test Problems

    Extract 2-3 specific problems from your work that you can test models against.

    Done when

    You have 2-3 concrete problems with example inputs and what good output looks like.

  2. Test Models on Your Problems

    Run 2-3 open models against your test problems and score their performance.

    Done when

    You have performance scores for each model-problem combination, based on your judgment of output quality.

  3. Calculate Real-World Costs

    Measure cost and speed for your test scenarios and compare to your current solution.

    Done when

    You have per-task cost estimates for open models vs your current approach, including infrastructure costs.

    Capstone: you put together your Open Model Evaluation Report.

Where this helps

An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.

About this lesson

What you'll be able to do

  • Test 2-3 open-source models on your actual work problems and document performance gaps
  • Generate cost-per-task comparisons between open models and your current solution
  • Make go/no-go decisions on open model adoption with evidence
  • Build a reusable testing framework for evaluating future models on your problems

Who it's for

  • Teams currently using paid AI services who want to evaluate cost savings
  • Technical leads exploring open-source alternatives for specific use cases
  • Product managers who need evidence-based decisions about model switching

What it won't cover

  • Building custom agent frameworks from scratch
  • Infrastructure setup and deployment pipelines
  • Fine-tuning or training models
  • Theoretical model architecture comparisons

Skills you'll practise

  • Comparative model evaluation
  • Business problem decomposition for AI testing
  • Cost-benefit analysis with real metrics
  • Performance benchmarking on domain tasks