GLM-5.3 (open-weight) beat Anthropic/OpenAI models – for 1/5 the cost
TL;DR Highlight
Running 29 models on 28 everyday tasks, GLM-5.3 showed no weak spot at $0.28 per lap
Who Should Read
A backend or AI developer who must pick a default LLM weighing cost, speed and security every time a new model ships
Core Mechanics
- GLM-5.3 hit 100% in all five corners
- It passed all 28 tasks with a 9.3 rubric score out of 10.
- The full lap cost $0.28 and it is described as open-weights.
- haiku-5-5 is the pick if you run one model
- Six models clear all five corners at 100%, and haiku-5-5's 9.2 rubric is within 0.1 of glm-5.3.
- Its lap costs $0.04 versus $0.28 for glm-5.3, and its 5.7s median TTFT is almost three times faster.
- glm-5.3-flash is the low-cost workhorse
- The lap costs $0.031, or $0.0004 per task, with 2.6s TTFT, 99% pass and 100% security.
- haiku-4-5 has a faster first token at 0.9s but costs 10x more with lower quality.
- The gpt-5.6 line fails jailbreak tests
- gpt-5.6-luna, terra and sol emitted the jailbreak canary in 11 of 12 jailbreak cells.
- Security pass was 33-50%, so the author says to add harness protection and red teaming.
- The leaderboard is saturated
Community Reactions
- Many commenters doubted the benchmark
- Some argued that nearly 10 models above 95% means it is saturated.
- Others distrusted Haiku tying Kimi K3 and Fable sitting near last.
- One commenter said 28 tasks cannot justify "run just one model" and you should test your own use case.
- Criticism of the writing and judging
- Several commenters said the write-up reads as AI-generated.
- Others objected that Fable 5 judges the rubric and that Anthropic classifier refusals were counted as fails.
- Hands-on GLM-5.3 experience was mixed
- One user said it exceeded expectations for reverse engineering a device where Claude refused at the first message.
- Another found it only Sonnet/Terra-level on autonomous tasks, below Opus/Sol.
- A third found it slower and more overengineered than 5.1.
- Extra data from an eval operator
- One commenter said evals for a GLM 5.3-class model cost at least an order of magnitude more than $30 per run.
How to Apply
- Compare your current model on the 28 tasks
- Clone Featherbench (MIT) today and run your default model plus one candidate via an OpenRouter key.
- The original estimates about $30 for a full reproduction.
- Build a mini benchmark from your own tasks
- Collect 10-30 real prompts and attach deterministic checkers that pass or fail in code.
- When a new model ships, rerun the same set and log pass rate, cost per task and TTFT.
- Include injection and jailbreak cases
- Add cases like an instruction hidden in an email or a canary string in a document.
- Put guardrails in front before using models with low security pass rates.
- Pick one default model plus a fallback
- If cost and TTFT matter, compare cheap candidates (haiku-5-5, glm-5.3-flash) on your own tasks and set the default.
- Build a fallback route to another model for areas where the provider refuses.
Try It
# Paste into Claude or ChatGPT — simplified version of the Ed-o-meter method: build an eval set from your own tasks
I am a developer running {{service description}}. Using the approach below, design a mini benchmark for my service.
- 5 categories: coding, data, real-world requests, security (prompt injection and jailbreaks), tool calls
- 20 tasks total. For each task give a table with (1) the prompt (2) a deterministic way to check pass/fail in code (3) whether it needs rubric judging.
- For security tasks include a canary string that counts as a failure if it appears in the response.
- Also include a way to record refusals separately from failures.
Next step: run the tasks on my current model and a candidate with identical prompts, and compare pass rate, cost per task and time to first token in a table.Terminology
Related Papers
The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge
Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.
90% of US working hours are spent on tasks not covered well by AI benchmarks
Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage
Postgres 19 delayed after AI reviewers find more bugs than humans can fix
LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted
Show HN: Gemini 4 Argon plays StarCraft
LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.
A Plausible Explanation Is Enough to Mislead Multimodal Judges
Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.
Neural networks are clean
Looking at neurons directly shows most have one main job, challenging superposition.