GPT-5.6 Sol can beat Factorio on peaceful mode
TL;DR Highlight
GPT-5.6 Sol in Codex at low thinking finished a peaceful Factorio run with a rocket launch
Who Should Read
Developer who hands long-running tasks to agents and is thinking about harness design and how to evaluate models
Core Mechanics
- Sol beat peaceful-mode Factorio
- It finished in 13.5 hours of game time in Codex at low thinking level.
- This was a peaceful run with no biters.
- A game manager gave a Python library
- The model played the game programmatically through a small library.
- Human GUI limits such as reach and fog of war were enforced.
- Scaling problems hit late
- The scaling issues the author expected at blue science appeared later.
- They slowed Sol's progress significantly.
- Rocket launched via manual ferrying
- Sol failed to connect some production systems.
- It repeatedly carried supplies between areas to cover the gaps.
- Better than GPT-5.5
- Earlier GPT-5.5 attempts failed to use transport belts at all.
- The author says Factorio resembles programming but is easier for casual observers to follow.
Community Reactions
- Opinion: lots of chests, little belt use
- A commenter noted that the core mechanic, belts, was used only in a few places.
- Most items sat in chests and were moved by the player.
- Opinion: Sol treated the engineer as automation
- A commenter saw a trace around 12 minutes cancelling a conflicting courier hand-feeding script.
- They said this loophole is hard to close with a scripting layer.
- Opinion: peaceful mode and training data limit the result
- Commenters said peaceful mode removes response time as a factor and models have read every guide.
- They suspect a similar but different game might go badly.
- Experience: models lose quickly on Civ 6 Deity
- One commenter tried a few recent models on Civ 6 Deity and all lost fast.
- Interest in the harness itself
- One commenter suggested using it as a gardener for long-running multiplayer bases.
- Another asked about blueprint support.
How to Apply
- Build a game-like test environment for agents
- Pick a simulation with clear rules to see if your agent handles long tasks.
- Enforce human-like limits such as reach and visibility in code.
- Log the agent's workarounds
- Scan run logs for patterns of repeated manual work instead of the intended method.
- Add how the goal was reached to your checks, not just whether it was.
- Compare models on the same harness
- Change only the model and thinking level, keeping the harness and prompt fixed.
- Collect time-series metrics like production and compare by milestone.
Try It
# Paste into Claude or ChatGPT — simplified version of the idea (checking agent workarounds)
You are an evaluator reviewing an agent run log. Find parts where the agent reached the goal through repeated manual work or workarounds instead of the proper method.
[Goal]
{{agent goal}}
[Run log]
{{log}}
For each workaround, make a table of (1) which proper method it failed to use (2) when it started (3) how the harness could block it.
Next step: run the same log through another model and compare the results.Terminology
Related Papers
AI isn't replacing SREs, it's deskilling them
Letting AI handle 95% of incidents may dull the skills SREs need for the critical 5%.
Backdoor in Claude Code for VS Code extension on the Cursor Marketplace
A tampered Claude Code extension on Cursor Marketplace planted an SSH backdoor
Building an AI-native engineering org
Automating each shipping step raised merges per engineer about 5x with reverts under 0.5%
We ran AI agent swarms in Minecraft and Mindustry
We put 50 agents in Minecraft and they invented ration ledgers and bureaucracy.
The Agent SDK
Cursor released Agent SDK, letting you define agents in code and deploy them with one command.
Here be monsters: What do you still need to know when agents do the work?
Owning agent-written code depends on fundamentals, breadth, and diving deep just in time.
AI promised less work. I've never worked more
Running agents in parallel brought my GitHub activity back to full-time engineer levels, and I'm exhausted.
Beyond Belief: Evaluating and Mitigating LLM Susceptibility to Misinformation
LLMs still ran harmful tool calls on 27–99% of false notices they rated untrustworthy.