
Evaluate open-source reasoning models on your actual business problems
You'll have a comparison document showing how frontier-class open models perform on 2–3 problems you actually need solved, with cost and latency metrics.
What you'll make
Open Model Evaluation Report
- Problem Test Suite: your actual tasks formatted for testing
- Model Performance Matrix: accuracy and quality scores per problem
- Cost Analysis: per-task costs vs current solution
- Deployment Decision: go/no-go with evidence
The 3 parts
Each part ends with a check you can see.
Define Your Test Problems
Extract 2-3 specific problems from your work that you can test models against.
Done when
You have 2-3 concrete problems with example inputs and what good output looks like.
Test Models on Your Problems
Run 2-3 open models against your test problems and score their performance.
Done when
You have performance scores for each model-problem combination, based on your judgment of output quality.
Calculate Real-World Costs
Measure cost and speed for your test scenarios and compare to your current solution.
Done when
You have per-task cost estimates for open models vs your current approach, including infrastructure costs.
Capstone: you put together your Open Model Evaluation Report.
Where this helps
An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.
About this lesson
What you'll be able to do
- Test 2-3 open-source models on your actual work problems and document performance gaps
- Generate cost-per-task comparisons between open models and your current solution
- Make go/no-go decisions on open model adoption with evidence
- Build a reusable testing framework for evaluating future models on your problems
Who it's for
- Teams currently using paid AI services who want to evaluate cost savings
- Technical leads exploring open-source alternatives for specific use cases
- Product managers who need evidence-based decisions about model switching
What it won't cover
- Building custom agent frameworks from scratch
- Infrastructure setup and deployment pipelines
- Fine-tuning or training models
- Theoretical model architecture comparisons
Skills you'll practise
- Comparative model evaluation
- Business problem decomposition for AI testing
- Cost-benefit analysis with real metrics
- Performance benchmarking on domain tasks