Prompts aren’t Real

TL;DR Highlight

Stop hand-tuning prompts; use pass^k tests and an optimizer to improve them automatically.

Who Should Read

Backend developer running a production agent that occasionally breaks schemas or rules and is nervous about editing prompts.

Core Mechanics

LLMs sometimes fail badly
- Even simple structured-output constraints like a title of 80 characters or less break in a fraction of requests.
- The smartest models do this too, and the failure rate shifts with the input, so you must keep watching.
Renaming a field fixed it
- A recent fix was renaming the field from "title" to "heading".
- The author expects such a deranged fix to break again at some point.
pass^k measures behavior
- Run the same test many times and check how consistently it passes (the industry term is pass^k).
- It helps catch changes that unintentionally break the agent.
A new prompt lands in a different universe
- The weight of the existing instructions changes how a new prompt performs, usually for the worse.
- So the pattern where the voice team owns the voice prompts doesn't scale.
An optimizer mashes the prompt for you

Community Reactions

A bring-your-own-AI opinion
- One commenter would prefer companies expose tools for the user's own AI rather than ship their own AI.
- That would sidestep the AI weirdness problem for the company.
Hit the same problems in production
- A commenter building an LLM production system said extensive tests were necessary.
- Stakeholders only see the few times something worked, not the long tail of cursed results.
Concern about test cost
- One commenter said running enough tests for statistical significance costs real money and the company might not survive it.
- Another called looping prompt optimization wasteful and said customers would stop paying if they did it.
Pushback that prompts still matter
- One opinion held that good constraints come from good prompts, which separate experts from beginners.
- A cynical summary said it's basically supervised learning on your prompts.

How to Apply

Write a pass^k test for your current prompt
- Pick the rule that breaks most often today and write a script that runs the same input 20 times.
- Record the pass rate as your baseline.
Have Claude generate adversarial and benign scenarios
- Give Claude the new skill or prompt and ask for abuse scenarios and normal scenarios that might break.
- Run the tests before and after the change and check that the pass rate improves.
Set aside holdout tests
- Hide a portion of the scenarios from both the optimizer and manual edits.
- If the holdout regresses after optimization, go back a step.
Build a golden dataset from production failures
- Sample real conversations, label the failures, and add them to the test suite.
- For complex judgments like brand voice, optimize a separate LLM judge using labeled data.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the article's method: test generation step only)

Below is the prompt (skill) I want to add to my agent.
{{prompt}}

For this prompt, create the following.
1. 10 scenarios where a malicious user tries to bypass these rules
2. 10 normal usage scenarios that this prompt might break

Output each scenario as JSON: {"input": user message, "expected": expected behavior, "check": how to judge (checkable in code or needs an LLM judge)}

Next step: run each scenario 20 times and compare the pass rate (pass^k) before and after applying the prompt.

Terminology

pass^kA metric that runs the same test k times and checks whether it passes every time. It shows that succeeding once differs from succeeding consistently.

GEPAA prompt optimizer where an LLM reflects on why a prompt did well or poorly on tests and revises it itself. It's like an automated reviewer that keeps patching in a loop.

Holdout testsTests kept hidden from the optimization process. They're the new exam questions that show whether the answers were memorized.

LLM judgeAn LLM that decides whether an agent's response met the criteria. It acts as the grader.

Golden datasetExpert-labeled good and bad responses used as a reference. It works as the ruler for quality.