How Specification Explicitness Shapes Failure in LLM-Generated Code
TL;DR Highlight
Spelling out invariants, edge cases and constraints cut hidden-test failures in LLM code by 47.2%.
Who Should Read
A backend developer whose LLM-generated code keeps breaking on edge cases despite reasonable-looking prompts.
Core Mechanics
- Structure alone barely helps
- The G2 template (goal, I/O, Given-When-Then criteria, constraints) averaged 19.67 failures versus 21.54 for G1.
- Filling in a form did not remove the gaps the LLM had to guess.
- Explicit behavioral obligations nearly halve failures
- G3 added invariants, negative requirements, edge cases, tie-breaking and exception handling.
- It averaged 11.38 failures, and distinct failure types fell from 4.04 to 3.21.
- Failures follow what the spec left unsaid
- Invariant violations fell from 92.71% to 31.46%, and unsupported assumptions from 58.33% to 23.61%.
- These were not random hallucinations but guesses filling unspecified behavior.
- Which elements you write matters
- Functional requirements, invariants and negative requirements were most associated with passing hidden tests.
- Acceptance criteria showed a negative association, which the authors read as a sign of harder tasks.
- Under change, regressions got more common but shallower
- G3 had regressions in 4/24 submissions versus 2/24 for G1 and G2.
- Each affected G3 submission lost less previously passing behavior.
- Explicit specs reduce overtrust
- G1 overestimated correctness by about 23 points.
- G3 wrote better code yet judged it conservatively, at -22.20 points.
Evidence
- G3 cut failures 47.2% versus G1
- Mean failures out of 33 hidden tests were 21.54 (G1), 19.67 (G2) and 11.38 (G3).
- G2 reduced failures by only 8.7% versus G1.
- Failure-type drops (G1 to G3)
- Invariant violations went from 92.71% to 31.46%, and constraint omissions from 76.49% to 51.36%.
- Unsupported assumptions went from 58.33% to 23.61%, and edge-case failures from 48.15% to 31.02%.
- Not just more prompting
- Adjusting for prompt count, length and type did not materially change the results.
- The adjusted G3 effect was not significant after Holm correction (IRR 0.48, pHolm=0.205).
- Regression depth, G3 versus G2
- Mean pass-to-fail transitions dropped from 0.38 to 0.17, a 55.3% reduction.
- Regression rate dropped from 2.38% to 0.70%, a 70.6% reduction, though only eight regressions occurred and the difference was not significant.
How to Apply
- Add invariant and prohibition fields before code generation
- Add slots for invariants, forbidden behavior, edge cases, tie-breaking and exception handling to today's prompt.
- Compare the output with a run of the same task without those slots.
- Make the LLM ask about missing obligations first
- Add a step that asks the model to list unspecified constraints and assumptions before coding.
- Generate code only after you fold the answers into the spec.
- Turn each spec item into a test
- Map invariants and negative requirements to tests one by one.
- Run them on each generation to catch violations early.
- Update spec, prompt, tests and code together on every change
- When a requirement changes, edit the spec first and paste it back into the prompt.
- Check that no stale invariants remain.
Try It
# Paste into Claude or ChatGPT as-is (simplified version of the paper's method)
You are a reviewer who checks the spec before any code is written. Do not write code from the requirements below yet. First fill in these items.
- Inputs/outputs and ordering rules
- Functional requirements
- Constraints
- Invariants (things that must always hold)
- Negative requirements (behaviors that must not happen)
- Edge cases and boundary conditions
- Tie-breaking/priority rules
- Exception handling
Original requirements:
{{requirements}}
For anything missing or ambiguous, do not guess; ask me questions first.
Once I answer, write the code and tests based on the completed spec, and label which spec item each test comes from.
Next to try: change one requirement, update the spec first, then regenerate the code.Terminology
Related Papers
Same Task, Different Statement: When and How Reformulation Helps LLM Problem Solving
Rewriting problem statements to fit their type raises LLM accuracy, but blind rewriting hurts.
Show HN: STE80 Humanizer – de-AI text by writing 80% of the way to ASD-STE100
A skill strips 26 AI tells and rewrites text in short ASD-STE100-style sentences
Character-Name Concentration in LLM Fiction: Measuring Displacement and Mitigating Repetition
Drafting name candidates and filtering by frequency cut LLM character-name repetition by 95%
LLM Voice
Martin Fowler: LLM-drafted prose keeps an uncanny tone even after human editing.
The Instruction Tax on Factual Recall: Base Models Can Beat Their Instruction-Tuned Counterparts
Few-shot completion with repeated sampling on a base model beat chat: F1 0.684 vs 0.568
AI is a tool and using it well takes skill
A rebuttal to Hickey: LLMs are tools, and skill in directing and checking them decides the outcome
Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfitting
Reflective prompt optimization raised held-out citation reward from 0.702 to 0.781 on a deep-research agent.
GPT‑6 and Intelligent UI for everyone
GPT-6 reportedly adds generated UI to chat; with no article text, only comment reactions are available.
Related Resources
Original Abstract (Expand)
When LLMs turn requirements into code, every underspecified constraint becomes behavior that the model must decide during generation. We test when specifications move beyond structure and begin to control generated behavior. In a controlled experiment, 72 participants implemented and revised Python tasks with an LLM under three levels of specification explicitness. The results show a clear break between lightweight structure and explicit behavioral obligations. Lightweight structure reduced hidden-test failures by only 8.7%. Requiring participants to specify constraints, invariants, negative requirements, edge cases, tie-breaking rules, and exception handling cut failures by 47.2% and reduced the number of distinct failure modes by 20.5%. The largest drops were in failure modes tied to implicit obligations. Invariant violations fell by 66.1%, and unsupported assumptions fell by 59.5%. After a requirement change, specifications with explicit behavioral obligations produced regressions in more submissions, but each affected submission lost less previously correct behavior. Explicit specifications also changed how developers judged generated code. Without explicit obligations, participants overestimated generated-code correctness by about 23 percentage points. With explicit obligations, they produced better code but judged it more cautiously. Thus, the lesson is sharper than "better specifications help." Lightweight structure barely helped, while explicit behavioral obligations nearly halved failures, made regressions shallower, and improved calibration.