Skip to main content
Skip to content
Menu

Claude Intermediate 3 parts Updated

Build a prompt-testing script for refusal patterns

You'll have a working Python script that logs Claude's responses to 10-15 edge-case prompts in your domain, flagging inconsistent refusal behavior so you can adjust your prompts.

What you'll make

Refusal Pattern Testing Script

  1. detect_refusal() — the pattern matching logic: what to include
  2. test_prompt_variants() — systematic edge-case testing: what to include
  3. analyze_inconsistencies() — comparison and flagging: what to include
  4. main() — orchestration with quantified results: what to include

Paste your Python script with comments showing your refusal detection logic and test results

The 3 parts

Each part ends with a check you can see.

  1. Build Your Refusal Detector

    Test 15-20 prompts against Claude by hand and code a detect_refusal() function based on actual response patterns you observe

    Done when

    A working detect_refusal() function that correctly identifies refusal responses in at least 90% of test cases, plus documented edge cases where detection fails

  2. Test Edge Case Variants

    Build test_prompt_variants() that systematically tests similar prompts and logs where Claude refuses inconsistently

    Done when

    Working test_prompt_variants() function that tests at least 5 prompt variant groups, plus documented inconsistencies where similar prompts get different refusal outcomes

  3. Measure Your Prompt Fixes

    Complete the script with analyze_inconsistencies() and measure how prompt modifications reduce false refusals in your test cases

    Done when

    Complete working script with quantified before/after results showing at least 3 prompt modifications that reduced false refusals, plus analyze_inconsistencies() function that flags similar patterns

    Capstone: you put together your Refusal Pattern Testing Script.

Where this helps

An elective: a single lesson that sits outside rtwo's job certificates and doesn't count toward one.

About this lesson

What you'll be able to do

  • Run a working Python script that tests your edge-case prompts against Claude and logs inconsistent refusal patterns
  • Identify 3-5 specific prompt modifications that reduce false refusals in your domain
  • Have a documented refusal detection algorithm that distinguishes legitimate safety responses from inconsistent behavior
  • Know how to systematically test AI boundaries rather than guessing at refusal causes

Who it's for

  • Developers building applications that use Claude for content generation or analysis
  • Anyone who's hit Claude's refusal walls and wants to map the boundaries systematically
  • Technical users who need predictable AI behavior for production workflows

What it won't cover

  • Jailbreaking or circumventing legitimate safety guardrails
  • Setting up Claude API credentials or Python environment basics
  • Building production-grade error handling or retry logic
  • General prompt optimization for better output quality

Skills you'll practise

  • Refusal pattern detection in API responses
  • Systematic boundary testing methodology
  • Edge-case prompt modification techniques
  • Quantitative measurement of AI behavior changes