GPT-5.6 vs. Claude Fable 5 for Physical AI, which performs best?
TL;DR Highlight
With the harness pinned across 52 graded runs, Fable scored 0.889 at 3–8x the cost of GPT models.
Who Should Read
A developer using agents for simulation or engineering code who is weighing model choice against cost.
Core Mechanics
- Fable won, but cost the most
- claude-fable-5 had the highest weighted score at 0.889.
- At $9.60 per trial, it cost 3–8x the GPT models.
- Sol was the value pick
- gpt-5.6-sol scored 0.814 at $1.74 per trial.
- Terra was cheapest and fastest at 0.786 and $1.25, while Luna trailed on every axis at 0.727.
- Easy problems were near perfect, P5 broke everyone
- All four models scored 1.000 on P1.
- On the hardest problem, P5 (HL-20 flight vehicle), Fable scored 0.690, Sol 0.660, Terra 0.590 and Luna 0.383.
- Grading used simulated trajectories, not code
- They picked problems that allow solutions that compile and run but are physically wrong.
- Scores come from comparing against sealed ground-truth trajectories.
- Only the model varied
- The Dyad harness, reasoning effort (xhigh), 1M context and 128k token budget were fixed.
Community Reactions
- Opinion that the benchmark is only partial
- A commenter noted effort levels weren't tested across a spread for each model.
- They argued there is no reason to believe Fable's xhigh equals GPT-sol's xhigh.
- Opinion that the cost is excessive
- A commenter said Fable cost $124.76 for marginal gains over the $22.56 Sol run.
- Opinion that comparisons are missing
- Commenters said Opus 5, Kimi 3 and GPT 5.5/5.4 were missing, making 0.786 vs 0.889 hard to judge.
- Reaction that it promotes their own harness
- One comment called it yet another benchmark promoting the author's own harness.
- Another noted Codex was missing from the harness comparison.
How to Apply
- Add ground-truth trajectory grading to your agent task
- Within 30 minutes, pick one representative task and write a script comparing output against a hidden expected result.
- Score on actual output match, not on whether tests pass.
- Pin harness, effort and budget when comparing models
- Keep prompts, tools, effort and token limits fixed while swapping models.
- Run each problem at least three times and average.
- Log score, cost and time together
- Record score, dollar cost and wall-clock time per trial.
- Consider defaulting to a value model and routing only hard tasks to the expensive one.
- Don't let the agent write its own tests
- Add verification independent of the agent for important work.
- Feed deliberately wrong inputs to confirm the check can fail.
Try It
# Paste into Claude or ChatGPT as-is (simplified version of the article's method: independent verification and deliberate breakage)
You are an agent that writes simulation code. Solve the problem below.
{{problem spec}}
Rules:
1. Before writing code, list the invariants (conserved quantities, constraints) that must hold physically.
2. Build an independent implementation (a different way of computing) and compare results at multiple points.
3. Confirm your verification can actually fail. Flip the sign of one input on purpose and show the residual grows.
4. Use the initial conditions in the spec as given without reinterpreting or renormalizing. If you changed any, say which value and why.
At the end, summarize the invariant checks and any differences from the spec in a table.
Next: run the same problem on another model and compare score, cost and time.Terminology
Related Papers
The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge
Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.
90% of US working hours are spent on tasks not covered well by AI benchmarks
Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage
Postgres 19 delayed after AI reviewers find more bugs than humans can fix
LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted
Show HN: Gemini 4 Argon plays StarCraft
LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.
A Plausible Explanation Is Enough to Mislead Multimodal Judges
Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.
Neural networks are clean
Looking at neurons directly shows most have one main job, challenging superposition.