I built a low-latency AI companion that plays Skyrim with me

TL;DR Highlight

Using a small classifier instead of an LLM to pick actions lets a Skyrim dog companion reply in under 500ms.

Who Should Read

An agent developer struggling with slow responses who wants to build real-time voice interaction products.

Core Mechanics

Existing AI NPCs talk well but act poorly
- The author says current frameworks are strong at dialogue and staying in character but weak at acting in the world, with high latency.
- They are also weak at complex multi-step instructions.
ALE picks actions instead of an LLM
- ALE (Action Latent Encoder) mixes embeddings, small classifiers, explicit rules and traditional ML.
- It detects negation, commands, pronouns and sequences, so "pick up the sword and bring it to me" becomes two linked action slots.
- It also takes the game's world state JSON as input and matches it to the request.
ALE came close to big LLMs while being fast
- It runs in about 2-20ms on an M4 MacBook.
- In the author's limited internal evals it performed surprisingly close to large LLMs at action selection and plan decomposition.
- The author does not consider this a serious benchmark.
The latency budget is split by stage
- Speech-to-text takes ~40-80ms, audio generation ~20-60ms, action analysis ~20ms.

Community Reactions

Commenters want details on ALE's design
- One asked what data pipeline from Skyrim was needed to get this action flexibility.
- Others asked whether multiple commands are split into sentences before embedding, and whether prototypes and classes are hand-coded or generated.
A view that GPT-Live could replace ALE
- One commenter said that if GPT-Live reaches the API at acceptable pricing, it would solve voice latency and ALE might be unnecessary.
- They noted it can dispatch tools asynchronously while speaking, which suits games.
- They said it is currently only a ChatGPT feature, not available to developers.
Experience that LLM quirks fit the character
- One commenter said the "dumb but lovable sidekick" persona makes LLM stumbles and the user's corrections feel like part of the game.
Comparison with an existing NPC chat mod
- One commenter who tried Mantella (talk to all NPCs via LLM) found the latency too high to enjoy, and said this approach looks much faster.

How to Apply

Measure LLM dependence with a classification prompt
- Pull 20 real user utterances today and have an LLM label each as command, question, chat, clarification or complex.
- Use the codeExample prompt and compare with ground truth to see how much a small classifier could replace.
Separate action selection from response generation
- Handle tool-call decisions first with embeddings, rules and a small classifier.
- Give the LLM only the chosen action to write the persona-flavored reply.
Measure a latency budget per stage
- Log milliseconds for STT, action selection, response generation and TTS.
- Shorten the slowest stage first with streaming or prefill.
Validate replies against world state before speaking
- Check that generated replies match items and targets in the current state JSON.
- Fall back to a cached reply on mismatch, or regenerate if time remains.

Try It

copy & run

# Paste into Claude or ChatGPT as-is (simplified version of the ALE-style action decomposition)

You are an action analyzer for a game companion. Look at the world state JSON and the player utterance below and answer with JSON only.

World state:
{{world_state_JSON}}

Player utterance:
{{utterance}}

Output format:
{"turn_type": "command|question|chat|clarification|complex", "actions": [{"verb": "...", "target_id": "an id from the world state or null", "depends_on": null}], "negation": false, "needs_clarification": false, "clarification_question": null}

Rules: Normalize synonyms (pick up, grab, fetch) to one verb. Resolve pronouns from the previous conversation. Never invent targets not in the world state; set needs_clarification to true instead.

Next: run 20 utterances, compare with ground truth, and see whether the failing types can be moved into rules or a classifier.

Terminology

ALE (Action Latent Encoder)A hybrid classifier that turns utterances into game action slots. Like a receptionist who splits an order slip into menu items and sequence.

groundingChecking that a model's reply matches the actual game state. Like verifying the item mentioned is really in the room.

VADVoice activity detection. A sensor that catches when a person starts and stops speaking.

prefillProcessing the input before the model writes its answer. Like opening the book before the question ends.

barge-inWhen someone interrupts mid-speech. Important so the companion doesn't talk over the player.