Show HN: Parsimony – an open-source benchmark for how much code AI models add
TL;DR Highlight
Counting syntax units in SWE-bench patches ranks which AI models add the least code
Who Should Read
A backend developer worried that bloated AI-agent code is making the codebase harder to maintain
Core Mechanics
- Benchmarks ignore code bloat
- Existing benchmarks measure only solve rate and cost.
- Excess code that makes a codebase harder to work with isn't penalized.
- It counts syntax elements, not lines
- It counts coding units such as calls, names and literals.
- Net units = added − removed.
- It only analyzes already-published patches
- Each patch is applied and both versions of every changed file are parsed.
- Tests aren't rerun, tests/docs/generated files are excluded, and only Python is measured.
- Ranking is mean net units on solved tasks, lowest first
- Solved status comes from SWE-bench Verified.
- All models use mini-SWE-agent, but the agent still affects how much code gets written.
- Smaller isn't always better
- Units measure size, not readability or design.
Community Reactions
- Astra winning is a weak signal, one commenter said
- They called Astra's code nearly unreadable code golf.
- They'd rather have Claude's excessive comments and tests.
- Add cyclomatic complexity as a supporting metric, one commenter suggested
- They see AI-written code increasingly suffering from complexity.
- They proposed it as a readability proxy.
- One commenter doubted that less code means less review work
- The metric doesn't show whether size tracks review effort.
- One commenter wanted local models included
- They want to see how close open-weight models are to commercial ones.
How to Apply
- Add a minimal-change rule to your agent prompt
- Add a minimal-change instruction to your coding agent's system prompt or CLAUDE.md.
- Run the same 3 issues before and after and compare diff size.
- Log net diff size per PR
- Record added and removed lines for each agent-made PR, by model.
- Compare per-model averages on the same tasks when choosing a default model.
- Pair code size with other review criteria
- Don't pick models on code size alone; check test passes and readability too.
- Filter out short but unreadable code-golf output separately.
Try It
# Paste into Claude Code or ChatGPT — simplified version of the Parsimony idea (minimal-change review)
Review the diff below. Decide whether this change is the minimum needed to resolve the issue.
{{diff}}
Answer in this order.
1. Separate necessary changes from unnecessary ones.
2. Point out unnecessary additions (duplicate logic, excess defensive code, needless helpers) line by line.
3. Show a smaller fix that keeps the behavior. Don't compress into unreadable code.
Next to try: compare added/removed line counts before and after the fix.Terminology
Related Papers
The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge
Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.
90% of US working hours are spent on tasks not covered well by AI benchmarks
Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage
Postgres 19 delayed after AI reviewers find more bugs than humans can fix
LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted
Show HN: Gemini 4 Argon plays StarCraft
LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.
A Plausible Explanation Is Enough to Mislead Multimodal Judges
Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.
Neural networks are clean
Looking at neurons directly shows most have one main job, challenging superposition.