GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2
TL;DR Highlight
GLM-5.2 scores 28% hallucination vs GPT-5.5's 86% on AA-Omniscience, the author says.
Who Should Read
A backend or agent developer choosing an LLM API and worried about hallucinations
Core Mechanics
- Bigger isn't always better, author argues
- GLM-5.2 (753B, ~40B active) is within 4 points of GPT-5.5 and 9 of Fable 5 on the Artificial Analysis Intelligence Index.
- The author says GPT-5.5 and Opus 4.8 are estimated at 1-2T parameters.
- Hallucination numbers vary widely
- On AA-Omniscience: DeepSeek V4 Pro 94%, GPT-5.5 86%, Fable 5 48%, Opus 4.8 36%, GLM-5.2 28%.
- DeepSeek V4 Pro said it didn't know only about 6% of the time on questions it couldn't answer.
- Longer reasoning didn't help
- On the same asyncio question, DeepSeek V4 Pro used about 7.7k reasoning tokens and gave a confidently wrong answer.
- GLM-5.2 spotted the impossible requirement in about 12 seconds and 799 tokens.
- The test is a single prompt
- Both models ran at high reasoning effort, temperature 1, on OpenRouter at FP8 precision.
- The system prompt was one line: "You respond professionally...".
- The author wants three things weighed together
- Model choice should balance capability, uncertainty calibration (hallucination rate), and compute efficiency.
- Don't pick by size or theoretical performance alone.
Community Reactions
- Commenters say the rate is conditional on not knowing
- It doesn't measure how often you'll meet a hallucination in daily use.
- One commenter computed absolute hallucination at 19% for Opus 4.8 and 21% for GLM-5.2.
- Commenters dispute the size explanation
- One noted DeepSeek-V4 Flash is under half GLM-5.2's size yet tops the hallucination index.
- Several said training method matters more than size.
- Commenters note the metric favors always abstaining
- A model that always says "I don't know" would score the lowest hallucination rate.
- Another shared that GPT-5.5 (xhigh) got 57% correct vs GLM-5.2 (max) at 25%.
- Real-world experiences were mixed
- One commenter said GLM 5.2 subtly changed requirements, building a Dioxus app when asked for Rust-WASM-Canvas.
- Another said GPT-5.5 was worse than Codex-5.3 and used 10x more tokens.
How to Apply
- Run a 20-question "I don't know" test on your domain
- Today, write 20 unanswerable or contradictory questions and run candidate models with identical settings.
- Record how often each abstains or flags the contradiction.
- Put hallucination and accuracy side by side in model comparisons
- Don't rely on benchmark score alone; tabulate accuracy, hallucination rate, latency, and token cost.
- Note that the hallucination rate is conditional on unknown questions.
- Add an impossibility check to the system prompt
- Tell the model to check whether requirements conflict or are impossible before answering.
- If so, have it explain why and propose alternatives.
Try It
# Paste into Claude or ChatGPT. Simplified version of the article's method: impossible-requirement detection test
You are a Python coding assistant.
Before writing code, check whether the requirements below conflict or are technically impossible.
If impossible, don't write code; explain why and suggest a realistic alternative.
If you don't know something, say "I don't know" instead of guessing.
Requirements:
{{requirements}}Terminology
Related Papers
Launch HN: Momentic (YC W24) – Agents that catch bugs by using your app
Mo, an AI QA agent that explores your app, found about 269 labeled bugs on QA Bench versus 86 for the next best.
We're releasing the weights of Mistral Large 4
Mistral released Large 4 weights, claiming top open-weight scores on coding benchmarks
Gemini 4 Carbon is the new #1 on the Artificial Analysis Intelligence Index v5
Gemini 4 Carbon takes #1 on Intelligence Index v5 with 59 and is second in coding agents.
Gemini 4 Carbon
Google unveils Gemini 4 Carbon, a cybersecurity-focused model, but access is limited to select cyber defenders.
Dayjob: Tax – evaluating AI agents on 50 professional tax assignments
On 50 professional tax tasks, the best AI agent passed only 10.8%
Anthropic S-1
Anthropic filed an S-1 to list Class A shares on Nasdaq as 'ANT'.
A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis
A fine-tuned Inception v3 beat GPT-6 Astra and Claude Fable 5.1 on 200 skin lesions
Why Does Structured Output Fail? An Empirical Study of Prompt-Enforced Pydantic Schemas Across Nine LLMs
Breaking structured-output failures into types showed markdown fences as the top cause, often hiding length violations.