LLMs Play Zachtronics' TIS-100
TL;DR Highlight
16 LLMs played TIS-100 and set 133 new Pareto frontier points across 18 puzzles
Who Should Read
LLM app developers who want to improve agent performance by adding code execution and feedback loops
Core Mechanics
- 16 models tackled 55 puzzles
- Each model got one scored attempt per puzzle (pass@1), with 10 to 45 minutes depending on difficulty.
- The community leaderboard and exact records were hidden from the models.
- 133 new frontier points on 18 puzzles
- A new frontier point is a solution that no community solution ties or beats on cycles, nodes, and instructions all at once.
- 18 of them beat a community record outright, and the other 115 are new tradeoffs.
- Opus 5.5 and GPT-6 Astra tie at the top
- Both solved all 55 puzzles and found new frontier points on 11.
- Opus 5.5 found more points, 72 to 32, while GPT-6 Astra had higher record proximity, 78.0 to 77.2.
- Solving and optimizing are different skills
- Gemini 3.8 Flash solved 53 puzzles but found new points on only 1.
- Fable 5.1 solved 52 and found new points on 5.
- Cost doesn't track quality
- GPT-6.1 Sol reached 75.6 record proximity at an estimated $0.75 per puzzle.
- Opus 5.5 reached 77.2 at $5.08, 7 times the cost for 2 more points.
- Fable 5.1 cost the most at $7.19 and scored 70.2.
- Speed is the hardest score
- Mean proximity was 72 for nodes, 56 for instructions, and 45 for cycles.
- New points appeared on 14 hard and 4 medium puzzles, and none on easy ones.
- Diagnostics helped every model
- In a separate six-puzzle study, models got target records, cycle-level profiles, a 10,000-test simulator, and up to 60 minutes.
- All five models improved mean record proximity, and scores rose in 28 of 30 model-puzzle pairs.
- Information, tools, and time changed together, so the study can't say which helped most.
Community Reactions
- One commenter found it interesting how poorly models did even though solutions are widely available online
- They were surprised by the gap.
- A commenter asked how human leaderboards for Zachtronics-like games can survive
- The worry is whether human rankings remain possible now that LLMs can solve optimization puzzles.
- A commenter said TIS-100 was mind-expanding but they stopped around puzzle 20
- They wished it had a web or TUI sandbox port.
How to Apply
- Return failure diagnostics to your code-running agent
- If your code execution tool only returns "failed", also return mismatched outputs, registers, and blocked ports.
- The model's next edit is then based on evidence.
- Build a get_best tool that saves the top result
- For agent tasks that iterate, store each passing result and allow recovery at any time.
- A risky edit then can't lose earlier progress.
- Add a reminder when the model quits early
- If the model tries to end early, send only a reminder to keep testing.
- Do not include solution hints.
- Compare models by score per cost
- Log cost and score per model on your own eval set.
- As in this benchmark, the pricier model isn't always better.
Try It
# Paste into Claude or ChatGPT (simplified version of the paper's method: diagnostics-driven iteration)
You are a code optimization agent. Solve the {{problem}} below.
Rules: first produce a passing solution, then improve it by changing one thing at a time.
At each step, record:
- The current best solution and its scores (speed, memory, code length)
- Which bottleneck this edit targets
- If it fails, which input broke it
{{problem}}
Next to try: at each step I'll paste {{run results or profile}}, and you propose the next edit based on that evidence.Terminology
Related Papers
Claude was trained on protein sequences at scale
Next-amino-acid tests on obscure proteins show only the latest Claude models predict them
Remembered as Brief: Stored Preferences SilentlyRemove Decision-Critical Information
A stored 'keep it short' preference silently drops warnings; listing must-know points first restores content.
All AI-generated code should be Open Source
A blog argues AI-generated code should be open source; most HN commenters disagreed.
What thousands of conversations look like inside a language model
Mapping Qwen3-1.7B states showed Chinese sitting apart from English, while German and Croatian overlapped it.
AI, activity, and outcomes in software engineering
AI sped up writing code, but shipping speed did not rise as much, says a talk
AI usage generates a fragile codebase
Generating all code with AI left shallow tests, complex code, and less human understanding, says one dev.
Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations
One release-note edge input cut wrong LLM repairs passing the gate from 0.410 to 0.065
Qwen 3.8 27B on my laptop can interpret model weights, too
Local Qwen 3.8 27B identified 3 of 4 games from raw network weights alone