Show HN: Jevman – AI decision models play Pac-Man | AI Paper Digest

TL;DR Highlight

Open-source Pac-Man benchmark ranks decision models over 100 real-time games each

Who Should Read

Backend developer weighing small decision models instead of chat LLMs for latency-sensitive routing or control logic

Core Mechanics

Each model plays 100 games
- Models face the classic scripted ghosts until all three lives are lost.
- Games are capped at 5 minutes, but the longest lasted only 2 minutes 24 seconds.
Models answer with odds per direction
- At each junction the game sends the maze, pellets and ghosts as JSON.
- The model returns a probability per direction and Pac-Man picks from them.
Answers over 2 seconds fall back to a rule
- A simple backup rule replaces late answers and counts them as backup moves.
- Claude Haiku 5.5, run by the authors, was left off because two thirds of its answers came late.
Ranking uses mean score with margin of error
- It uses the mean score with a 95% margin (±2 standard errors).
- Models within each other's margin are tied, and the high score is the best single game.
Text-writing models are not ranked

Community Reactions

Opinion that the model should run every tick, not just at junctions
- The commenter notes Pac-Man can change direction at any point.
Author update that community contributions doubled the tested models
- A new model took #1 and over 1000 games have been played against decision-model ghosts.
Question about local CPU models and a commenter's experience training small models
- One commenter trained small models to play Doom and plans to train one for this benchmark.
Opinion that a regular LLM reference would help
- Others said mobile controls were hard and joked about paying 2 cents per game.

How to Apply

Log latency against a 2-second limit on one decision point
- Pick one place in your service where an LLM makes a decision and record response time today.
- Check what share of calls exceeds 2 seconds.
Add a rule-based fallback to the LLM call
- Set a timeout and let a simple rule decide when it is exceeded.
- Track how often the fallback fires.
Try a per-option probability output format
- Make a classification or routing prompt return only JSON probabilities per option, with no explanation.
- Compare latency and accuracy against your current text answers.
Compare scores by mean and margin of error
- When comparing models A and B, run at least 100 trials and compute the mean and ±2 standard errors.
- Treat overlapping ranges as tied.

Try It

copy & run

# Paste into Claude/ChatGPT — simplified version of the article's method (probabilities-only decision prompt)

You are a decision model. Output JSON only, with no explanation.
Situation: {{situation description}}
Options: {{option list, e.g. up, down, left, right}}
Give the probability that each option is best, summing to 1, in the format {"up":0.0,"down":0.0,"left":0.0,"right":0.0}.

Next step: run the same situation 20 times and check how much the probabilities vary and whether responses arrive within 2 seconds.

Terminology

Decision modelA model that returns only probabilities per option in one pass instead of writing text. Like a test taker who only fills in odds on an answer sheet.

Backup moveA simple rule used when the model's answer arrives too late. It plays the role of a default value on timeout.

Margin of errorThe range showing how much an average score may fluctuate. If two models' ranges overlap, you can't say one is really better.

JunctionA fork in the maze where Pac-Man can choose a direction. The game asks the model at each one.

Show HN: Jevman – AI decision models play Pac-Man