Skip to main content
Skip to content
Menu

AI Vulnerability Intermediate 4 parts

Build a classifier script that flags AI-generated bug reports in your codebase

You'll have a working script that analyzes incoming bug reports and assigns confidence scores for AI origin, flagging reports that need human verification before triage.

What you'll make

AI Bug Report Classifier

  1. classify_report() — the main LLM-based scoring function: what to include
  2. load_patterns() — your documented signature patterns from real data: what to include
  3. route_decision() — confidence threshold logic and escalation rules: what to include
  4. main() — file processing and output formatting: what to include

Paste your complete Python script with example usage on real report files

The 4 parts

Each part ends with a check you can see.

  1. Collect Your Ground Truth

    Gather 50+ bug reports from your project and hand-label them as likely human vs likely AI-generated based on your domain knowledge

    Done when

    You have 50+ reports with confident human/AI labels and at least 10 examples where you're genuinely uncertain

  2. Build the Detection Logic

    Write classify_report() and load_patterns() functions that score reports for AI likelihood based on the patterns you identified

    Done when

    You have a working classifier that produces confidence scores (0-100) and correctly identifies at least 70% of your hand-labeled training data

  3. Set Your Confidence Rules

    Test different confidence thresholds against your data to find the right balance between catching AI reports and avoiding false positives

    Done when

    You have documented threshold rules (e.g., >80 = auto-flag, 50-80 = human review, <50 = normal triage) with measured false positive and false negative rates

  4. Package the Working Script

    Complete main() with file processing and output formatting, then test the full script on a fresh batch of bug reports

    Done when

    You have a complete script that processes report files, outputs routing decisions with confidence scores, and includes usage instructions for other maintainers

    Capstone: you put together your AI Bug Report Classifier.

Where this helps

An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.

About this lesson

What you'll be able to do

  • Run a working classifier script on at least 50 real bug reports from your project
  • Identify the 3-5 linguistic patterns where AI-generated reports differ systematically from human-written ones
  • Have a documented confidence threshold and escalation rule for reports that need human review
  • Know which report features to trust vs which require verification regardless of AI confidence score

Who it's for

  • Maintainers of open-source projects getting increasing volumes of bug reports
  • Security teams triaging vulnerability reports from mixed human and AI sources
  • Anyone managing a bug tracker who's noticed patterns in report quality but can't articulate what's different

What it won't cover

  • Determining if a bug report describes a real vulnerability (that's separate triage)
  • Training custom ML models from scratch (this uses LLM APIs for classification)
  • Blocking AI submissions entirely (this is about intelligent routing, not gatekeeping)

Skills you'll practise

  • Pattern recognition in text classification
  • Confidence threshold calibration from real data
  • Human-AI handoff design
  • False positive cost analysis