Terminal-Bench-Science: Evaluating AI agents on scientific research workflows
TL;DR Highlight
On 70 scientist-written tasks, the best agent resolves only 30%
Who Should Read
Developers who feel coding benchmark scores alone are not enough when choosing models and harnesses for agents
Core Mechanics
- 70 tasks written by scientists
- Stanford researchers built it with the Terminal-Bench team.
- It covers life, physical, earth, mathematical, and engineering sciences.
- Top score is only 30%
- Claude Opus 5 on Claude Code leads at 30.0%.
- Next are GPT-5.6 Sol on Codex at 22.4% and Claude Fable 5 on Claude Code at 21.4%.
- Deliberately made hard
- Every model evaluated on both scores more than 10 points lower than on Terminal-Bench 3.0.
- Tasks were calibrated during review to challenge the newest frontier models.
- Graded by tests on artifacts
- Analyses, simulations, proofs, code, and data are checked by reproducible task-specific tests.
- Each model ran three independent trials per task.
- Cost and tokens differ from rank
- GPT-5.6 Sol matches Claude Fable 5 at $4.2k vs $14.2k.
Community Reactions
- Opus 5 beating Fable 5 seems odd
- One commenter said Opus 5 feels worse than Fable for coding.
- Another also found the gap strange.
- Worry that correctness isn't checked
- One user asked Claude to implement a paper's algorithm and got a simpler, slower one, then just an apology when challenged.
- They said it can't be trusted for paper-grade work.
- Sol's math lead was welcome
- A commenter was glad Sol beats Opus in mathematical sciences.
- Math is their current need and they prefer GPT's prose.
- Fear the benchmark saturates quickly
- A commenter expects big score jumps with next releases, making it useless.
- They wished it were more closed.
How to Apply
- Write 3 of your domain tasks as eval cases
- Spend 30 minutes today picking three recurring jobs for your agent.
- Write the input, expected output, and an automatic pass criterion for each.
- Build an eval that grades artifacts with tests
- Write scripts that check files, code, or numbers instead of reply text.
- Run each task 3 times and average the resolution rate.
- Add cost and tokens to model selection
- Record total cost and token usage next to resolution rate.
- When performance is similar, pick the cheaper or lower-token model.
Try It
# Paste into Claude or ChatGPT (simplified version of the method: turn a job into a verifiable eval task)
Job my agent does: {{job description}}
Example input: {{input data}}
Turn the job above into one Terminal-Bench-Science style eval task.
1) Task description (the instruction given to the agent)
2) Output files and format the agent must produce
3) A Python pytest that automatically grades the output
4) Ways to tune difficulty so current top models still struggle
Next to try: run the same task through your agent 3 times and count the passesTerminology
Related Papers
The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge
Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.
90% of US working hours are spent on tasks not covered well by AI benchmarks
Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage
Postgres 19 delayed after AI reviewers find more bugs than humans can fix
LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted
Show HN: Gemini 4 Argon plays StarCraft
LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.
A Plausible Explanation Is Enough to Mislead Multimodal Judges
Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.
Neural networks are clean
Looking at neurons directly shows most have one main job, challenging superposition.