MTG Bench: Testing how well LLMs can play Magic
TL;DR Highlight
Without a rules engine, gpt-5.5 medium led an LLM Magic turn benchmark at 95.4
Who Should Read
A backend developer running agent loops over an MCP server who worries about per-model cost and tool-call mistakes
Core Mechanics
- No rules engine, the LLM runs the turn
- The author assumed an LLM good enough to play Magic does not need a rules engine.
- The MCP server only exposes primitive library operations like draw, return to bottom, and shuffle.
- Complex actions like scry or surveil are built by the model from multiple tool calls.
- gpt-5.5 medium scored highest
- gpt-5.5 medium scored 95.4, claude-fable-5 medium 90.3, and gpt-5.4 medium 88.6.
- claude-opus-4-8 medium scored 39.8 and deepseek-v4-pro high scored 12.8.
- Effort mattered: gpt-5.5 low scored 84.5 versus 95.4 at medium.
- More expensive did not mean better
- Cost per turn was $0.10 for gpt-5.5 medium, $0.30 for claude-fable-5 medium, and $0.19 for claude-opus-4-8 medium.
- claude-opus-4-8 cost more than gpt-5.5 medium yet scored 39.8.
- gpt-5.4-nano scored 68.2 at $0.01.
- Eager tool calls can't be undone
- If you draw a card and return it, you have already seen it, so the simulation is illegal.
- A common failure was starting a tool call, realizing the mistake, and having no way to fix it.
- Every library function has a required reason field, and Opus 4.8 used reasons like 'noop' and 'stop' while returning a card named 'x'.
- gpt-5.5 medium did the grading
- Legality checks and scoring were all done with gpt-5.5 (medium).
- In the author's testing, LLMs were much better at judging whether a turn was legal than at performing a legal turn.
- Remote MCP billing differs by provider
- OpenAI charges the input prompt once for a single API call using a remote MCP server.
- Anthropic charges the cached system prompt input again after every tool use.
- Average input tokens per turn were 11,386 for gpt-5.5 (medium) and 51,610 for claude-fable-5 (medium).
Community Reactions
- Opinion: pit models against each other with a rules engine
- One commenter said counting illegal moves and wins or unfinished games would be better, and that LLM grading seems unreliable.
- Another wondered about adapting Card Forge for LLM use.
- Experience running with the xMage rules engine
- One commenter had LLMs play the Elves vs Goblin duel decks on xMage and found it too expensive, so they could only afford deepseek models.
- They said the models sometimes make serious blunders and the harness is hard to build.
- Experience with local models on mage-bench
- One commenter ran a 4-model local tournament on an RTX 5090 and said Qwen 3.6 27B narrowly beat Gemma 4.
- Concerns about grader bias and naming
- One commenter asked whether gpt-5.5 as grader could favor models that think like it.
- Another said it only measures rule-following, not how well models play, since there is no opponent.
How to Apply
- Make a reason field required on tool calls
- Add a required string reason field to the MCP tool schemas you use today.
- Start with state-mutating tools and check logs for meaningless reasons like 'noop'.
- Add a check before irreversible tools
- List tools whose effects can't be undone, such as payments, deletes, or secret reads.
- Add a step in the prompt that reviews the plan once before calling them.
- Measure agent loop cost per provider
- Log input token usage for requests with remote MCP on OpenAI and Anthropic separately.
- If your system prompt is large, compare cost by tool-call count before picking a model.
- Separate verification from execution
- Pass agent action logs to a separate model call that checks for rule violations.
- Use a different model for verification than for execution to reduce bias.
Try It
# Paste into Claude or ChatGPT (simplified version of the article's method: separate-call legality check)
You are a referee for a card game simulation. Below is the log of a turn an agent ran by calling tools.
Rules:
- A card you have seen counts as revealed information even if you put it back in the deck.
- If an irreversible tool call is followed by a cancel or restart, it is illegal.
- Check that each call's reason matches the actual action.
Log:
{{turn log}}
Output format:
1. Verdict: legal / illegal
2. If illegal, the call number at fault and why
3. Whether the agent self-reported its mistakeTerminology
Related Papers
Gemini 4 Carbon
Google unveils Gemini 4 Carbon, a cybersecurity-focused model, but access is limited to select cyber defenders.
Dayjob: Tax – evaluating AI agents on 50 professional tax assignments
On 50 professional tax tasks, the best AI agent passed only 10.8%
Anthropic S-1
Anthropic filed an S-1 to list Class A shares on Nasdaq as 'ANT'.
A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis
A fine-tuned Inception v3 beat GPT-6 Astra and Claude Fable 5.1 on 200 skin lesions
America has lost 200k of jobs in media, film and the arts. How much is AI?
US creative-industry jobs fell by over 200k in four years, but AI's share of the blame is hard to prove
No signs of AI payoff outside the tech sector
Revenue per worker and margins rise at the Mag 7 but stay flat across the S&P 493
LLMs are making formal software verification feasible at a much larger scale
Claude proved no-crash properties for 123 open-source libraries and found bugs along the way.
PHP2Shell: AI uncovers critical RCE in PHP
Wiz used AI to turn a decade-old PHP core flaw into a working RCE exploit