13 Models and 4 Agents on SWE Tasks: Go, Java, Python, Rust, TS
TL;DR Highlight
On 111 fresh GitHub issues, Fable 5 led at 64.5% resolved, with costs varying widely
Who Should Read
A developer choosing a coding model or agent who wants to compare resolve rate against cost
Core Mechanics
- It re-tests on recent issues
- The window is 2026-05-15 to 07-01, with 111 problems from 65 repositories.
- The leaderboard lists both models and agents (Junie, Claude Code, Codex, Cursor).
- The top is clustered at 63-65%
- Resolved Rate: Fable 5 [high] 64.5%, Grok 4.5 [high] 63.8%, Opus 5 [high] 63.4%.
- GLM-5.2 [high] is at 62.9% and GPT-5.6 Sol [medium] at 62.3%.
- Agents sit at ranks 6-8
- Junie is 6th at 61.8%, Claude Code 7th at 60.4%, Codex 8th at 58.0%.
- Cursor is 10th at 51.7%.
- Cost doesn't follow rank
- First-place Fable 5 costs $4.40 per problem, while GPT-5.6 Sol [medium] costs $0.85.
- Their resolved rates are 64.5% vs 62.3%.
- Claude Code costs $3.39 and Cursor $0.41.
- Cheap models stand out
Community Reactions
- A comment noting problem sets differ
- One commenter said problems differ across languages.
- They had hoped for a benchmark showing which languages work best with which models.
- A question about Claude Code setup
- A commenter asked why Claude Code is 7th while Fable 5 and Opus 5 are 1st and 3rd.
- They wanted to know which model and effort the 7th-place entry used.
- Doubts about the effort comparison
- A commenter asked why Fable high was compared with Sol medium.
- They noted Sol comes out 4-5x cheaper.
- Contamination warning and field experience
- A comment quoted the About page on public SWE-bench issues possibly being in training data.
- Another commenter said budget models worked much better for them than this test suggests.
How to Apply
- Run candidate models on 5 of your own issues
- Pick 5 recent bug issues from your repo today.
- Give the same prompt to one or two top models and one cheap model.
- Record resolved or not, plus token cost, in a table.
- Pick by resolve rate and cost per problem
- Put Resolved Rate and Cost per Problem side by side when choosing a model.
- For example, decide between 62.3% at $0.85 and 64.5% at $4.40 based on your budget.
- Check your caching setup first
- Verify prompt caching is on in your agent calls.
- Top models on the leaderboard show around 90% of tokens as cached.
Try It
# Paste into Claude or ChatGPT (simplified version of the idea: build a model comparison plan)
I'm choosing a model for a coding agent. Using the benchmark numbers below, draft a small comparison experiment to run on our repo.
[SWE-rebench numbers]
- Fable 5 [high]: resolved 64.5%, $4.40 per problem
- GPT-5.6 Sol [medium]: resolved 62.3%, $0.85 per problem
- MiMo V2.5 Pro: resolved 46.5%, $0.10 per problem
[Our constraints]
- Language: {{language}}
- Monthly budget: {{budget}}
- Recent bug issues: {{5 issues}}
Request: 1) steps to compare the three models on 5 issues 2) metrics to record (resolved, tokens, cost) 3) a table of decision criteria.
Next: paste your results and ask which model to use as the default.Terminology
Related Papers
The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge
Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.
90% of US working hours are spent on tasks not covered well by AI benchmarks
Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage
Postgres 19 delayed after AI reviewers find more bugs than humans can fix
LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted
Show HN: Gemini 4 Argon plays StarCraft
LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.
A Plausible Explanation Is Enough to Mislead Multimodal Judges
Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.
Neural networks are clean
Looking at neurons directly shows most have one main job, challenging superposition.