Why Does Structured Output Fail? An Empirical Study of Prompt-Enforced Pydantic Schemas Across Nine LLMs

Oct 24, 2026•Amit Verma•View PDF

TL;DR Highlight

Breaking structured-output failures into types showed markdown fences as the top cause, often hiding length violations.

Who Should Read

A backend developer who puts a Pydantic schema in the prompt, runs a validate-and-retry loop, and can't tell why outputs keep failing.

Core Mechanics

A single validity rate hides the cause
- Models with similar failure rates had different dominant failure categories.
- A rate alone can't tell you whether to fix the parser, the schema, or the retry feedback.
Markdown fences are the top failure
- markdown_wrap was 681 of 1,126 failures (60.5%).
- Only 48.0% passed validation after the fence was stripped.
Fences hide length violations
- The other 52.0% had a further violation inside, mostly field length.
- The parse error returned to the model never mentions it.
Raising length caps removes length failures
- For Sonnet 4.6, length_bound failures went from 18 to 0 after raising the caps.
- regex_pattern then became the dominant category.
One regex mixes two causes
- ^[a-z][a-z0-9_]{2,30}$ encodes both the character class and a length ceiling.
- Loosening only the character class cut leading-digit violations from 4 to 0, but length overflows remained.
The same model fails differently by domain
- Sonnet 5 had similar failure rates on both domains, but fences dominated on HotpotQA.
- On FinanceBench no single category clearly dominated.

Evidence

Haiku 4.5 failed every first attempt
- All 475 first attempts were fenced and failed strict validation.
- Its attempt failure rate was 66.1% and only 142 of 166 runs completed.
Fence-strip revalidation results
- Only 327 of 681 fenced failures (48.0%) passed.
- The rest were 327 length_bound, 26 regex_pattern and 1 missing_field.
Relaxing caps tested on Sonnet 4.6
- Revalidating 178 stored medium outputs against medium_v2 made all 18 length failures pass.
- No valid output became invalid, and the failure rate fell from 15.7% to 5.6%.
32.2% of retry messages missed the hidden violation
- 354 of 1,101 retry messages named only a parse error.
- 350 of them were Haiku 4.5, and all 24 run-nodes that exhausted retries ended on length.

How to Apply

Log a fence-strip revalidation
- On validation failure, strip the ```json fences and run model_validate_json again.
- If it passes, record formatting-only; if not, record the remaining error.
Separate length limits from regexes
- If a quantifier like {2,30} enforces length, move that to min_length/max_length fields.
- Your logs will then distinguish length violations from character-class violations.
Set caps from real output lengths
- Measure actual output lengths for free-text fields like answer and notes first.
- If caps sit below them, check if it's a real downstream requirement, then raise the cap or instruct length in the prompt.
Build retry feedback from the fence-stripped validation
- Validate the fence-stripped payload before composing the retry message.
- The paper did not test this, so measure the change in retries yourself.
Report failure composition per configuration
- Aggregate failure types per model and schema instead of one overall rate.
- Count concealed violations too.

Try It

copy & run

# Run in Python: simple diagnostic to tell fence-only failures from hidden violations (simplified version of the paper's method)
# Install: pip install pydantic

import re
from pydantic import BaseModel, Field, ValidationError

class Answer(BaseModel):
    answer: str = Field(min_length=20, max_length=400)

def strip_fence(raw):
    s = raw.strip()
    s = re.sub(r"^```[a-zA-Z]*\n?", "", s, count=1)
    return re.sub(r"\n?```\s*$", "", s, count=1)

def diagnose(raw):
    try:
        Answer.model_validate_json(raw)
        return "valid"
    except ValidationError:
        pass
    if raw.lstrip().startswith("```"):
        try:
            Answer.model_validate_json(strip_fence(raw))
            return "pure_formatting"
        except ValidationError as e:
            return "compound: " + e.errors()[0]["type"]
    return "other"

long_text = "x" * 500
print(diagnose('```json\n{"answer": "' + long_text + '"}\n```'))

# Next: feed your own logged responses {{raw_output}} into diagnose and tally by type

Terminology

PydanticA Python data-validation library. You define a schema class and it checks whether incoming JSON follows the rules.

markdown_wrapA failure where the model wraps JSON in ```json code fences, breaking strict parsing. Like receiving a package still in its box when you need the contents.

constrained decodingA method that only allows grammar-conforming tokens during generation. This paper does not study it, only prompt-and-validate.

compound failureA failure where a length or pattern violation remains after the fence is removed. Like unwrapping a gift and finding the item also defective.

question-clustered bootstrapA resampling method that groups attempts by question to compute confidence intervals. Used when data like retries are not independent.

Related Resources

Original Abstract (Expand)

Pipelines that require structured outputs from hosted language models often specify a Pydantic schema in the prompt, validate the response, and retry when validation fails. Although native structured-output modes provide an alternative, application-side validation remains important for enforcing custom constraints. Yet pipeline reliability is often summarized by a single validity rate, which obscures why outputs fail and which interventions may help. We introduce a fine-grained taxonomy of schema-conformance failures and a deterministic classifier that assigns failure categories from model responses and Pydantic validation errors. We apply this framework to 3,997 generation attempts across nine OpenAI and Anthropic model/API configurations, using questions from HotpotQA and FinanceBench. Our results show that similar validity rates can mask different failure patterns, and that dominant failure categories vary with schema constraints and task. Markdown-fenced JSON is the most common failure: among 681 fenced failures, only 48.0% pass validation after fence removal, while the remainder conceal additional violations, predominantly field-length violations. In an exploratory prompt ablation, variants that suppress fences appear to shift failures toward length violations, but the shift is consistent with exposing violations that were already present, illustrating how aggregate validity rates can obscure changes in failure composition. We derive four engineering practices for validator, schema, retry and evaluation design, three supported by our data and one, on retry feedback, proposed but not tested, and release the classifier, completions, and annotations.

Why Does Structured Output Fail? An Empirical Study of Prompt-Enforced Pydantic Schemas Across Nine LLMs

Oct 24, 2026•Amit Verma•View PDF

TL;DR Highlight

Breaking structured-output failures into types showed markdown fences as the top cause, often hiding length violations.

Who Should Read

A backend developer who puts a Pydantic schema in the prompt, runs a validate-and-retry loop, and can't tell why outputs keep failing.

Core Mechanics

A single validity rate hides the cause
- Models with similar failure rates had different dominant failure categories.
- A rate alone can't tell you whether to fix the parser, the schema, or the retry feedback.
Markdown fences are the top failure
- markdown_wrap was 681 of 1,126 failures (60.5%).
- Only 48.0% passed validation after the fence was stripped.
Fences hide length violations
- The other 52.0% had a further violation inside, mostly field length.
- The parse error returned to the model never mentions it.
Raising length caps removes length failures
- For Sonnet 4.6, length_bound failures went from 18 to 0 after raising the caps.
- regex_pattern then became the dominant category.
One regex mixes two causes
- ^[a-z][a-z0-9_]{2,30}$ encodes both the character class and a length ceiling.
- Loosening only the character class cut leading-digit violations from 4 to 0, but length overflows remained.
The same model fails differently by domain
- Sonnet 5 had similar failure rates on both domains, but fences dominated on HotpotQA.
- On FinanceBench no single category clearly dominated.

Evidence

Haiku 4.5 failed every first attempt
- All 475 first attempts were fenced and failed strict validation.
- Its attempt failure rate was 66.1% and only 142 of 166 runs completed.
Fence-strip revalidation results
- Only 327 of 681 fenced failures (48.0%) passed.
- The rest were 327 length_bound, 26 regex_pattern and 1 missing_field.
Relaxing caps tested on Sonnet 4.6
- Revalidating 178 stored medium outputs against medium_v2 made all 18 length failures pass.
- No valid output became invalid, and the failure rate fell from 15.7% to 5.6%.
32.2% of retry messages missed the hidden violation
- 354 of 1,101 retry messages named only a parse error.
- 350 of them were Haiku 4.5, and all 24 run-nodes that exhausted retries ended on length.

How to Apply

Log a fence-strip revalidation
- On validation failure, strip the ```json fences and run model_validate_json again.
- If it passes, record formatting-only; if not, record the remaining error.
Separate length limits from regexes
- If a quantifier like {2,30} enforces length, move that to min_length/max_length fields.
- Your logs will then distinguish length violations from character-class violations.
Set caps from real output lengths
- Measure actual output lengths for free-text fields like answer and notes first.
- If caps sit below them, check if it's a real downstream requirement, then raise the cap or instruct length in the prompt.
Build retry feedback from the fence-stripped validation
- Validate the fence-stripped payload before composing the retry message.
- The paper did not test this, so measure the change in retries yourself.
Report failure composition per configuration
- Aggregate failure types per model and schema instead of one overall rate.
- Count concealed violations too.

Try It

copy & run

# Run in Python: simple diagnostic to tell fence-only failures from hidden violations (simplified version of the paper's method)
# Install: pip install pydantic

import re
from pydantic import BaseModel, Field, ValidationError

class Answer(BaseModel):
    answer: str = Field(min_length=20, max_length=400)

def strip_fence(raw):
    s = raw.strip()
    s = re.sub(r"^```[a-zA-Z]*\n?", "", s, count=1)
    return re.sub(r"\n?```\s*$", "", s, count=1)

def diagnose(raw):
    try:
        Answer.model_validate_json(raw)
        return "valid"
    except ValidationError:
        pass
    if raw.lstrip().startswith("```"):
        try:
            Answer.model_validate_json(strip_fence(raw))
            return "pure_formatting"
        except ValidationError as e:
            return "compound: " + e.errors()[0]["type"]
    return "other"

long_text = "x" * 500
print(diagnose('```json\n{"answer": "' + long_text + '"}\n```'))

# Next: feed your own logged responses {{raw_output}} into diagnose and tally by type

Terminology

PydanticA Python data-validation library. You define a schema class and it checks whether incoming JSON follows the rules.

markdown_wrapA failure where the model wraps JSON in ```json code fences, breaking strict parsing. Like receiving a package still in its box when you need the contents.

constrained decodingA method that only allows grammar-conforming tokens during generation. This paper does not study it, only prompt-and-validate.

compound failureA failure where a length or pattern violation remains after the fence is removed. Like unwrapping a gift and finding the item also defective.

question-clustered bootstrapA resampling method that groups attempts by question to compute confidence intervals. Used when data like retries are not independent.

Original Abstract (Expand)

Why Does Structured Output Fail? An Empirical Study of Prompt-Enforced Pydantic Schemas Across Nine LLMs

TL;DR Highlight

Who Should Read

Core Mechanics

Evidence

How to Apply

Try It

Terminology

Related Papers

Related Resources

Why Does Structured Output Fail? An Empirical Study of Prompt-Enforced Pydantic Schemas Across Nine LLMs

TL;DR Highlight

Who Should Read

Core Mechanics

Evidence

How to Apply

Try It

Terminology

Related Papers

Related Resources