Show HN: A new benchmark for testing LLMs for deterministic outputs
TL;DR Highlight
Across 28 LLMs, JSON pass rates top 97% but value accuracy runs 15-30 points lower.
Who Should Read
Backend developer running an LLM pipeline that extracts JSON from invoices, transcripts, or PDFs into a database and choosing a model.
Core Mechanics
- JSON parsing no longer separates models
- Most modern models clear 95%+ JSON Pass on the unified leaderboard.
- Value Accuracy (share of leaf field values exactly matching ground truth) is 15-30 points lower.
- Structure metrics hit 99% while values are wrong
- Path Recall, Structure Coverage and Type Safety can read ~99% while 20-30% of leaf values are wrong.
- Perfect Response (every field right in a record) is around half even for top models.
- The leader is pricey but the gap is small
- Value Accuracy: Gemini-3.1-Pro 82.0%, GLM-5.1 80.6%, GLM-4.7 80.4%, Qwen3.5-35B 80.1%.
- Qwen3.5-35B, GLM-4.7 and Gemini-2.5-Flash cost 6-11x less per correct field than the leader, losing at most ~2 points.
- Winners differ by modality
- Best Value Accuracy: text 84.5% (Gemini-3.1-Pro), image 67.2% (Gemma-4-31B), audio 23.7% (Gemini-2.5-Flash).
- Image and audio were converted to text-normalized context before scoring.
- Newer generations barely improve
- GPT-5.5 trails GPT-5.4 by 0.3 points on unified Value Accuracy.
- Claude-Opus-4.7 beats Opus-4.6 by under 1 point.
- Plausible wrong values are the worst bug
- On an audio record, ground truth "15 to 35 years" came back as "25 to 35".
- The type and schema are valid, so only field-level checks catch it.
Community Reactions
- Solve first, format as JSON second
- One commenter reported much higher quality when the model first produced code and a separate call wrapped it in JSON, in an agentic state machine.
- They argued single-pass parse-and-format is fragile and wanted benchmarks with an intermediate step and a BAML comparison.
- Doubts about model selection and neutrality
- Commenters said missing frontier models like Opus 4.7 looked selective.
- One asked how a third party can verify neutrality since the vendor's own Interfaze is on the leaderboard.
- Wrong values, not parse errors, hit production
- One commenter said that at 97% JSON pass, nothing flags wrong values without a baseline.
- They said valid-but-incorrect data is what actually reaches production.
- Why not constrained decoding, and skepticism
- Someone asked why structured (constrained) decoding isn't used.
- Others called it a subset of hallucination benchmarks and said to avoid LLMs if you want determinism.
How to Apply
- Measure field accuracy on 50 real samples
- Today, take 50 real inputs and attach human-verified ground-truth JSON.
- Script per-field exact match instead of just JSON pass rate.
- A/B cheaper models on the same samples
- Run your current model and candidates like Qwen3.5-35B, GLM-4.7 or Gemini-2.5-Flash on the same set.
- Tabulate accuracy and cost per correct field to decide on switching.
- Try two-step extraction versus one call
- Build a flow that extracts freely first, then converts to JSON in a second call, and compare to a single call.
- Add source-quote fields to the schema
- Add an excerpt field for each critical field.
- Verify in code that the value actually appears in the excerpt.
Try It
# Paste into Claude/ChatGPT (simplified version of the SOB idea: field-level grounding check)
Extract JSON matching {{schema}} from {{document}} below.
Rules:
1. For each field, include the value and an evidence quote copied verbatim from the document.
2. If the document lacks it, set value to null and evidence to "". Do not guess.
3. Do not change the document's notation for numbers, ranges, or dates.
4. Output JSON only.
Then, as a separate step, check field by field whether each value exactly matches its evidence and list mismatches.
{{document}}Terminology
Related Papers
Gemini 4 Carbon
Google unveils Gemini 4 Carbon, a cybersecurity-focused model, but access is limited to select cyber defenders.
Dayjob: Tax – evaluating AI agents on 50 professional tax assignments
On 50 professional tax tasks, the best AI agent passed only 10.8%
Anthropic S-1
Anthropic filed an S-1 to list Class A shares on Nasdaq as 'ANT'.
A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis
A fine-tuned Inception v3 beat GPT-6 Astra and Claude Fable 5.1 on 200 skin lesions
America has lost 200k of jobs in media, film and the arts. How much is AI?
US creative-industry jobs fell by over 200k in four years, but AI's share of the blame is hard to prove
No signs of AI payoff outside the tech sector
Revenue per worker and margins rise at the Mag 7 but stay flat across the S&P 493
LLMs are making formal software verification feasible at a much larger scale
Claude proved no-crash properties for 123 open-source libraries and found bugs along the way.
PHP2Shell: AI uncovers critical RCE in PHP
Wiz used AI to turn a decade-old PHP core flaw into a working RCE exploit