Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
TL;DR Highlight
On 25 private enterprise tasks, 17 models were tested and the best resolved only 56.8%.
Who Should Read
A backend developer choosing models and harnesses for coding agents to adopt on an in-house codebase
Core Mechanics
- 17 models tested on private real-world code
- 25 tasks come from licensed production codebases of real companies.
- The codebases span TypeScript, Go, Python, Ruby and Java.
- Even the top model, Opus 5.5, resolves 56.80%
- Sonnet 5.5 is second at 51.20% and GPT 6.1 Sol third at 47.20%.
- The rate is pass@1 averaged over five runs per task, at the highest effort possible for each model.
- Unverified assumptions are the top failure
- Across all models, unverified assumptions account for 33.1% of failures and missed requirements for 29.4%.
- For GPT 6 Astra it is 55.1% of failures, and for GPT 6.1 Sol 53.0%.
- Cost varies widely
- Estimated cost per rollout ranges from $0.08 (GPT 6 Luna) to $14.24 (Fable 5.1).
- GPT 6.1 Sol hits 47.20% at $0.79, while Opus 5.5 hits 56.80% at $13.95.
- Tasks are short instructions with many-file changes
Community Reactions
- Commenter built the same kind of bench on own repos
- They cut git history just before a change, gave the ticket as the prompt, and graded against the accepted PR.
- Finding good examples took the most time.
- Commenter paired Fable with Sol as advisor
- Fable throws out unchecked assumptions and Sol verifies them and alerts Fable.
- Reversing the roles made Fable just say it sounds good.
- Opinion that private codebases may not be private
- A commenter suspects many large private codebases have already been exposed via Claude Code and Codex.
- Another reported measuring contamination every time when running a different bench.
- Pushback that task specs may be weak
- Without a human baseline, it is hard to separate model skill from task quality.
- The commenter reads about 60% failures from bad assumptions and missed requirements as underspecified tasks.
How to Apply
- Run a mini benchmark on one recent merged PR
- Check out the PR's parent commit and give the agent the original ticket as the prompt.
- Compare the result with the real PR and mark where it guessed.
- Add an assumption-check step to your prompt
- Make the agent list assumptions to verify and cite file locations before implementing.
- Pin this in the system prompt since unverified assumptions are the top failure.
- Compare resolution rate and cost together when picking a model
- Run the same tasks on a cheap and a top model and put both in a table.
- Start with the cheap model in a review role.
- Label failure types in your internal task set
- Classify failed runs as unverified assumption, missed requirement, wrong file and so on.
- Tune prompts and checklists to the most frequent type.
Try It
# Paste into Claude Code or ChatGPT (simplified version of the paper's method: assumption-check-first prompt)
You are an engineer modifying an existing codebase. Before implementing the requirements below, do the following.
1. Split the requirements into numbered items.
2. For each item, list the assumptions it depends on (function behavior, config values, external service contracts) and cite the file and line you checked directly. Mark unchecked assumptions as 'unverified'.
3. Verify unverified assumptions by reading or running code before implementing.
4. Look for existing code that already does the same thing (duplicate paths) and follow the company's existing coding patterns.
5. After implementing, check each item number for satisfaction and confirm with tests that existing behavior is not broken.
Requirements: {{ticket content}}
Next step: run the same ticket once more without this prompt and compare the number of missed requirements and wrong assumptions.Terminology
Related Papers
The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge
Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.
90% of US working hours are spent on tasks not covered well by AI benchmarks
Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage
Postgres 19 delayed after AI reviewers find more bugs than humans can fix
LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted
Show HN: Gemini 4 Argon plays StarCraft
LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.
A Plausible Explanation Is Enough to Mislead Multimodal Judges
Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.
Neural networks are clean
Looking at neurons directly shows most have one main job, challenging superposition.