“Next-token predictor” is the wrong mental model for LLMs | AI Paper Digest

TL;DR Highlight

RLVR trains LLMs to maximize reward, not predict data tokens, so next-token predictor is incomplete.

Who Should Read

A developer using LLM APIs who is unsure whether 'just a next-token predictor' is an accurate way to describe LLMs.

Core Mechanics

Inference still emits tokens one at a time
- At generation time the loop does look like a next-token predictor.
- The author says the claim is not wrong, just incomplete.
Pre-training really is next-token prediction
- It raises the probability of the token that actually followed in the training data.
- So the base model can be called a next-token predictor.
RLVR has no correct next token to predict
- RLVR is reinforcement learning with verifiable rewards.
- Tokens in sequences the model generated itself become more likely when those sequences earn high reward.
Same function, different reason
- Pre-training raises a token's probability because it appeared in the data.
- RLVR raises it because the sequence containing it earned a high reward.
A chess engine illustrates the gap

Community Reactions

Counterpoint that it is still prediction
- One commenter argued what is learned is still P(token_k | previous tokens).
- In their view where it was learned from does not change the task.
Opinion that pre-training is not plain next-token prediction either
- A commenter noted backprop sums loss contributions from all later tokens into each position's representation.
- So representations are not optimized only for the next token.
Experience that the next-token model explains real errors
- A commenter saw 'schmuck' output for 'schmutz' and answers locked in by an early 'Yes' that later got contradicted.
- They said the next-token model explained both.
Opinion that the chess analogy is weak
- A commenter noted chess engines pick a whole branch from a search tree.
- They said no one has explained where a transformer stores a future plan.
Opinion that the right model depends on context

How to Apply

Sort model failures into two layers
- Pick 3 failing prompts today and label each as early-token lock-in or failed reward-style problem solving.
- If the first answer is wrong, reorder so the model writes reasoning before the answer and rerun.
Reorder prompts to put the conclusion last
- In Yes/No judgment prompts, require reasoning first and the verdict at the end.
- Compare on the same inputs to see whether early tokens stop fixing the answer.
Attach a result verifier to checkable tasks
- For code or math, run tests and feed pass/fail back to the model for retries.
- This mirrors how RLVR uses pass/fail as the reward.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the paper's method: mimicking a verified reward)

You are a solver for the problem below. Solve it with 3 different approaches, then score each solution 0-10 using a verifiable criterion (tests or recomputation).

Problem: {{problem}}

Rules: put the conclusion only at the very end; write the reasoning and verification first. Choose the highest-scoring solution as the final answer.

Next to try: ask the same problem with "state the conclusion first" and compare accuracy.

Terminology

RLVRReinforcement learning where outcomes can be automatically verified and rewarded. It is like taking an exam where you only get the grade.

Pre-trainingThe first training stage where the model learns to predict the next token over huge text. Like reading enormous amounts and guessing what comes next.

Post-trainingStages after pre-training, like RLHF and RLVR, that shape model behavior. Like on-the-job training for a new hire.

RLHFTraining with human preference ratings as reward. The post says it shifts the model toward simulating a helpful assistant.

AutoregressiveGenerating one token at a time, feeding earlier outputs back in as input. Like a word chain game.

Related Papers

Related Resources

https://gmcgoldr.github.io/2026/09/04/llm-next-token-predictors.html