Skip to main content
SkillTycoonSign in

Lesson 1 of 6 · Stop prompting on vibes

Why a prompt that worked stops working

From The Prompting Playbook: Debug a Production Prompt: An interactive walkthrough of "The Prompting Playbook", a public breakout talk at the Code with Claude conference by Margot van Laar, Applied AI Engineer at Anthropic in London. Follow her live debugging of a messy customer-support prompt for a fictional telco: why a prompt that worked breaks on a new model, how to build an eval suite of control, edge and capability-boundary cases, how to strip pasted-website cruft and an untrue persona, how XML tags separate role from policy from tone, and how an output contract plus a stop sequence pin down the response — then isolate a single failing case and find the conflicting instruction behind it. Source: Margot van Laar, "The Prompting Playbook", Code with Claude.

Stop prompting on vibes · 01

Why a prompt that worked stops working

You migrate to a new model and suddenly a lot of your test cases regress. Nothing about the prompt changed — so what actually broke? This course follows a live debugging session as far as the recording goes — which stops mid-fix.
  • Source: “The Prompting Playbook” · Code with Claude
  • Talk by: Margot van Laar, Anthropic (London)
  • Format: Interactive — predict, sort, decide
The talk sets up two scenarios you probably recognise from work: an existing production prompt you have been maintaining that is suddenly performing worse, and a blank canvas where you have to build an agentic prompt from zero to one. This course follows the first one.

How the prompt got into this state

Four ordinary things that compound — tap each
  • No clear owner
  • Every concern in one place
  • Patches for old models
  • Then you migrate

Predict: what do you need to know first?

Predict

Your prompt regressed on the new model. Before you start rewording anything, what is the first question you have to be able to answer?

Quick check

Reading a regression
A prompt regresses after a model migration. Which of the two causes can prompting actually fix?
Which of these describes the state the example prompt is in before any work starts?

Topics: prompt-engineering, evals, llm, production, claude, anthropic, debugging, interactive