Can a MUD evaluate LLMs? A $99 proof of concept

TL;DR Highlight

Running 13 models in a text MUD showed one LLM judge component shifting rankings by up to six places.

Who Should Read

A backend developer building an agent evaluation pipeline with LLM judges who worries about score reliability

Core Mechanics

A MUD measures agent behavior
- CrucibleBench puts models in a text world where NPCs remember and trust accumulates.
- It scores how well they reach hidden social objectives over 50 turns.
Constraints make measurement easy
- With 7 command types, 12 rooms and 14 items, hallucinated actions are directly detectable.
- Four NPCs carry 0–100 trust and suspicion state, so you can see whether models adapt to feedback.
One LLM judge shifted rankings by six places
- Removing the two dimensions that depend on a dialogue classifier moved six rankings beyond sampling noise.
- Gemini 3.1 Pro fell from #3 on the full score to #9 under classifier-minimized scoring.
Aggregate reliability missed it
- Per-model agreement with an independent judge ranged from 21.7% to 84.8%, yet aggregate κ was 0.04.
- The biggest mover shared a model family with the classifier.

Community Reactions

MUDs work as an agent sandbox, per one experience
- One commenter let local LLMs play a live MUD and saw them build maps, classify events and verify grammars, exposing model strengths and weaknesses.
Adding human-in-the-loop tools, per one experience
- One commenter built a few-room MUD with an oracle tool and hint interjection, then limited turns to see where the agent got stuck.
Objection that it is not a real MUD
- One commenter said it is a bespoke MUD-style environment with no source code, so the headline is overstated.
Suggestion to measure combat and progression
- One commenter suggested measuring fighting, looting and stat progression, which need no server-side LLM.

How to Apply

Audit your current judge per model
- Pull 50 eval logs today and compute agreement with human labels or a different-family judge for each model.
- Check the per-model minimum, not just the overall average.
Compare scores with and without the judge
- Drop scoring items that rely on LLM classification and recompute the rankings.
- If rankings change a lot, report both versions together.
Add rule-based failure detection from logs
- Count repeated calls to one target, calls to nonexistent targets and consecutive lookups with simple rules, and fold them into the failure rate without any judge.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified prompt version of CrucibleBench failure detection)

You are an agent log auditor. Count failures using rules only, no judgment.
Rule 1: 8 or more talk-type commands at the same target = 'dialogue looping'.
Rule 2: acting on a nonexistent target or room that returns an error = 'wrong-target interaction'.
Rule 3: 2 or fewer rooms over 20+ turns, or 5 consecutive look commands = 'exploration paralysis'.

Log:
{{agent log}}

Output a table of the count per rule and the turn numbers.

Next step: run the same log through another model and compare the results.

Terminology

MUDMulti-User Dungeon. A text-based online game world from the early internet where you type commands to move between rooms and talk to NPCs.

LLM judgeAn LLM that grades another LLM's output. Like a referee who is also a player, its own biases leak into the scores.

Cohen's κA measure of how much two raters agree beyond chance. A low overall value can hide large differences in per-model agreement.

judge ablationRemoving the LLM judge part of scoring to see how scores and rankings change.

paired block bootstrapA statistical method that resamples data in blocks to test whether a rank change is more than chance.