Show HN: Gemini 4 Argon plays StarCraft
TL;DR Highlight
LLMs wrote StarCraft bots in one hour; GPT-6 Astra scored highest.
Who Should Read
An LLM developer whose coding-agent choices don't match what standard benchmark scores suggest.
Core Mechanics
- Each model gets one hour to write a bot
- Each LLM writes a Protoss bot in C++ within one hour.
- The score is the average win rate (0-100) against a roster of human-written bots.
- GPT-6 Astra ranks first
- It is clearly ahead of GPT-6.1 Sol in second.
- Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 are functionally tied at 37-39.
- Practicing against strong bots predicts score
- The strongest predictor is how many practice games a model played against the 7 strongest roster bots.
- Claude Fable 5.1 is the outlier, scoring 37 while practicing mostly against weaker bots.
- GPT-6.1 Sol is the value pick
- It scores second at just over a tenth of GPT-6 Astra's cost per run.
- It is the only model in the most attractive quadrant.
- Top human bots still win
Community Reactions
- Ranking matches GoBench, per one commenter
- A commenter said a 9x9 Go benchmark by a friend produces exactly the same model order.
- The problems differ but the ordering is the same.
- Concern that it measures training-data recall
- Some said many bot implementations on GitHub make this a recall-from-training-set test.
- Another asked how pre-training data use is avoided.
- Harness matters a lot, per one commenter
- They said the same model can perform very differently across harnesses, allowed tools and thinking levels.
- They also asked how a model reads documentation with web search turned off.
- Unclear what it really measures, per one commenter
- They listed code generation speed, logic analysis, game platform understanding and strategy as mixed factors.
- Opus 5.5 worked better in one experience
- One commenter building bots for a ModernUO server said GPT-6.1 Sol and GPT-6 Astra made little progress in a week.
How to Apply
- Compare coding agents on your own task
- Pick one real issue today and give it to 2-3 candidate models with the same time limit.
- Run each in its own first-party CLI harness and compare the results.
- Make agents practice against hard cases
- If your agent task has a test or eval loop, tier the cases from easy to hard.
- State in the prompt that hard cases must be run early.
- Run the same model several times to see variance
- For important agent tasks, run the same setup 5 times.
- Record the mean along with the best-worst gap.
- Pick models with your own roster, not generic scores
- Build a fixed roster of internal tasks separate from public leaderboards.
- Score each new model's win rate on the same roster.
Try It
# Prompt to paste into Claude Code or Codex (simplified version of the benchmark's method)
You must finish {{task: e.g. optimize the slow API endpoint in this repo}} within one hour.
Rules:
1. First list tests ordered from easy to hard: {{test list}}
2. Early on, run a hard test once to see where it currently fails.
3. Repeat a loop of fixing the cause of failures and rerunning.
4. When time ends, the code must always be in a buildable state.
5. After each iteration, log in one line what you practiced (tested) and what you fixed.
At the end, summarize the hard-test pass rate and what you fixed.
Next step: run the same prompt 3-5 times per model and compare the mean and spread of pass rates.Terminology
Related Papers
The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge
Warning an LLM judge about bias or letting it self-critique improved 20 of 21 model–bias combinations.
90% of US working hours are spent on tasks not covered well by AI benchmarks
Mapping 29 benchmarks to job tasks shows 90% of US working hours lack good coverage
Postgres 19 delayed after AI reviewers find more bugs than humans can fix
LLM code review surfaced so many bugs that Postgres 19 slipped to November and 8-10 features were reverted
A Plausible Explanation Is Enough to Mislead Multimodal Judges
Adding plausible explanations to fixed answers cut MLLM judges' wrong-answer rejection by 12.7–13.8 points.
Neural networks are clean
Looking at neurons directly shows most have one main job, challenging superposition.
Linus Torvalds's first LLM-assisted patch series
Linus posted an LLM-assisted 10-patch series giving an order-of-magnitude read() speedup, with a race already found.