
Design a one-page scorecard template measuring your engineering team's AI sophistication
You'll have a visual scorecard template with 4–5 dimensions (prompt quality, evaluation rigor, revision cycles, output trust) that moves past activity metrics to the behaviors that actually correlate
What you'll make
Engineering AI Sophistication Scorecard
- Scorecard Dimensions: the 4-5 behaviors that predict AI effectiveness
- Scoring Rubrics: 1-5 scale definitions for each dimension with observable criteria
- Data Collection: how to gather evidence for each dimension without surveillance
- Interpretation Guide: what combinations of scores mean and which gaps to address first
Write the complete scorecard as a reusable template with specific measurement instructions
The 3 parts
Each part ends with a check you can see.
Map Current AI Behaviors
Inventory how your team actually uses AI today — not what they should do, what they do.
Done when
You have documented evidence of 3-5 different AI usage patterns across your team with specific examples of verification vs acceptance behaviors.
Test Measurement Approaches
Try 2-3 different ways to score the behaviors you identified and see which ones work.
Done when
You have tested 3 measurement approaches on the same team examples and identified which scoring methods produce reliable, non-gameable results.
Build Scorecard Template
Assemble your tested dimensions and rubrics into a one-page scorecard you can actually use.
Done when
You have a complete one-page scorecard template with 4-5 dimensions, clear scoring rubrics, and specific instructions for data collection that takes under 30 minutes per team member.
Capstone: you put together your Engineering AI Sophistication Scorecard.
Where this helps
An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.
About this lesson
What you'll be able to do
- Run a scored assessment of your team's actual AI behaviors using 4-5 measurable dimensions
- Identify the 2-3 capability gaps that matter most for your team's AI productivity
- Have a one-page scorecard template you can rerun quarterly to track AI sophistication growth
- Know which team behaviors to reinforce vs which AI habits to redirect
Who it's for
- Engineering managers who see uneven AI adoption across their team
- Tech leads wondering if their team is using AI effectively or just burning tokens
- Directors who need to measure AI investment ROI beyond 'lines of code generated'
What it won't cover
- Individual performance reviews or compensation decisions
- Specific AI tool recommendations or procurement
- Code quality metrics or traditional engineering productivity measures
- Team restructuring or hiring based on AI skills
Skills you'll practise
- Behavioral pattern recognition in AI tool usage
- Scorecard design for emergent capabilities
- Gap analysis between AI activity and AI sophistication
- Assessment calibration across different engineering roles