Skip to main content
Skip to content
Menu

Evaluations Intermediate 3 parts Updated

Write an honest model-capability brief

You'll have a 1-2 page assessment of a specific AI model's real strengths and weaknesses in your domain, including tasks it should not touch and how you verified that claim.

What you'll make

Model Capability Brief

  1. Model Context: what you tested and why
  2. Verified Strengths: tasks where it consistently succeeds
  3. Systematic Failures: patterns where it breaks down
  4. Boundary Conditions: edge cases and limitations
  5. Deployment Recommendations: when to use vs avoid

Write as a 1-2 page brief you'd actually send to a colleague making a similar decision

The 3 parts

Each part ends with a check you can see.

  1. Test the Happy Path

    Run the model on 5 straightforward tasks from your domain and document what good performance looks like

    Done when

    5 completed test cases with concrete examples of the model's output and your assessment of quality

  2. Hunt for Systematic Failures

    Design tests specifically to reveal where the model breaks down consistently and document the failure patterns

    Done when

    At least 3 adversarial test cases that reliably produce poor output, with documented patterns of how the model fails

  3. Write the Capability Brief

    Synthesize your testing into a practical brief that documents verified capabilities and clear recommendations for use

    Done when

    A complete 1-2 page brief with specific use cases, documented failure modes, and actionable deployment recommendations

    Capstone: you put together your Model Capability Brief.

Where this helps

An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.

About this lesson

What you'll be able to do

  • Test a specific AI model on 3-5 real tasks from your work and document where it succeeds vs fails
  • Identify at least 2 systematic failure modes where the model consistently gets it wrong
  • Have a written brief you'd actually share with colleagues about when to use vs avoid this model

Who it's for

  • Anyone evaluating whether to adopt a specific AI model for their team or workflow
  • People who need to make honest recommendations about AI capabilities to colleagues or stakeholders
  • Professionals tired of vendor demos and marketing claims who want real capability data

What it won't cover

  • Technical model architecture or training details
  • Comparative benchmarking across multiple models
  • Cost-benefit financial analysis or ROI calculations

Skills you'll practise

  • Adversarial evaluation design
  • Failure mode documentation
  • Capability boundary identification
  • Evidence-based model assessment