
Build a prompt-testing script for refusal patterns
You'll have a working Python script that logs Claude's responses to 10-15 edge-case prompts in your domain, flagging inconsistent refusal behavior so you can adjust your prompts.
What you'll make
Refusal Pattern Testing Script
- detect_refusal() — the pattern matching logic: what to include
- test_prompt_variants() — systematic edge-case testing: what to include
- analyze_inconsistencies() — comparison and flagging: what to include
- main() — orchestration with quantified results: what to include
Paste your Python script with comments showing your refusal detection logic and test results
The 3 parts
Each part ends with a check you can see.
Build Your Refusal Detector
Test 15-20 prompts against Claude by hand and code a detect_refusal() function based on actual response patterns you observe
Done when
A working detect_refusal() function that correctly identifies refusal responses in at least 90% of test cases, plus documented edge cases where detection fails
Test Edge Case Variants
Build test_prompt_variants() that systematically tests similar prompts and logs where Claude refuses inconsistently
Done when
Working test_prompt_variants() function that tests at least 5 prompt variant groups, plus documented inconsistencies where similar prompts get different refusal outcomes
Measure Your Prompt Fixes
Complete the script with analyze_inconsistencies() and measure how prompt modifications reduce false refusals in your test cases
Done when
Complete working script with quantified before/after results showing at least 3 prompt modifications that reduced false refusals, plus analyze_inconsistencies() function that flags similar patterns
Capstone: you put together your Refusal Pattern Testing Script.
Where this helps
An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.
About this lesson
What you'll be able to do
- Run a working Python script that tests your edge-case prompts against Claude and logs inconsistent refusal patterns
- Identify 3-5 specific prompt modifications that reduce false refusals in your domain
- Have a documented refusal detection algorithm that distinguishes legitimate safety responses from inconsistent behavior
- Know how to systematically test AI boundaries rather than guessing at refusal causes
Who it's for
- Developers building applications that use Claude for content generation or analysis
- Anyone who's hit Claude's refusal walls and wants to map the boundaries systematically
- Technical users who need predictable AI behavior for production workflows
What it won't cover
- Jailbreaking or circumventing legitimate safety guardrails
- Setting up Claude API credentials or Python environment basics
- Building production-grade error handling or retry logic
- General prompt optimization for better output quality
Skills you'll practise
- Refusal pattern detection in API responses
- Systematic boundary testing methodology
- Edge-case prompt modification techniques
- Quantitative measurement of AI behavior changes