Skip to main content
SkillTycoonSign in

Lesson 1 of 8 · Measure before you change anything

Why a prompt that worked stops working

From The Prompting Playbook: Debug a Production Prompt: Your production prompt worked, then you moved to a new model and the test cases started failing. This is the repair method, in order. Build a small eval suite of control, edge and capability-boundary cases, so you can tell a real improvement from a feeling. Strip the pasted-website cruft and the untrue persona. Carve the prompt into tagged sections so role, policy, guidelines and tone stop competing. Add an output contract and enforce it in the harness with a stop sequence. Then isolate one failing case at a time and find the instruction that made the wrong answer the compliant one. Worked end to end on a messy customer-support prompt for a fictional telco, then run on a prompt of your own. Based on “The Prompting Playbook” by Margot van Laar, Code with Claude.

Measure before you change anything · 01

Why a prompt that worked stops working

You migrate to a new model and suddenly a lot of your test cases regress. Nothing about the prompt changed — so what actually broke? Here you debug a real production prompt the way an engineer does: measure first, clean, then fix one failure at a time.
  • Based on: “The Prompting Playbook” by Margot van Laar · Code with Claude
  • Format: Interactive — predict, sort, decide
There are two prompt problems you meet at work. One: a production prompt you maintain is suddenly performing worse. Two: a blank page, where you build an agentic prompt from nothing. This course takes the first one.

How the prompt got into this state

Four ordinary things that compound — tap each
  • No clear owner
  • Every concern in one place
  • Patches for old models
  • Then you migrate

Predict: what do you need to know first?

Predict

Your prompt regressed on the new model. Before you start rewording anything, what is the first question you have to be able to answer?

Quick check

Reading a regression
A prompt regresses after a model migration. Which of the two causes can prompting actually fix?
Which of these describes the state the example prompt is in before any work starts?

Topics: prompt-engineering, evals, llm, production, claude, anthropic, debugging, interactive

Why a prompt that worked stops working — The Prompting Playbook: Debug a Production Prompt · SkillTycoon