How to build a diffusion language model

TL;DR Highlight

A guide to diffusion LLMs: fill and refine whole sequences in parallel instead of one token at a time.

Who Should Read

An LLM API developer frustrated by autoregressive speed and lack of error correction who wants to understand how diffusion LLMs work.

Core Mechanics

Autoregressive LLMs have three limits
- A token cannot be revised once emitted, so early mistakes compound.
- Generation needs as many steps as tokens, so parallelism is hard.
- Causal attention never looks at future context.
Diffusion LLMs refine a whole draft
- They start from a full-length draft and rewrite every position in parallel over several steps.
- Fewer or more steps trade speed against quality, and each step sees bidirectional context.
- By 2026 there are Mercury 2 (Inception Labs), Gemma Diffusion (Google), and Nemotron Diffusion (NVIDIA).
Masked diffusion is a generative BERT
- Train a bidirectional transformer to fill in blanks at a random masking rate.
- Generation starts fully masked and repeats infilling and remasking.
- Unlike BERT the masking rate is randomized, and the post says this forms the basis of most open-source diffusion LMs.
The loss is an averaged BERT loss

Community Reactions

Running diffusion Gemma on GPUs was reported very fast in output tokens/sec and usable as a local model
- The commenter noted its whitepaper says it could have done better with more time and compute.
Reimplementing diffusiongemma exposed a weakness, per one commenter
- When two positions must be coordinated but both teeter between valid states, they may not collapse to a matching pair.
One commenter wondered about generating text as images with image diffusion, then OCRing it
- They guessed a single monospace font would make OCR close to template matching.
A university student said deriving the ELBO was illuminating
- Naming the larger math structures made it simple, and ChatGPT helped fill gaps in understanding.

How to Apply

Feed this post to an LLM to judge fit for your service
- Today, run the prompt below with your latency and error-correction requirements.
- Ask it to separate where parallel generation and iterative refinement help from where they do not.
Benchmark Diffusion Gemma speed on a local GPU
- Compare output tokens/sec against your current autoregressive model on the same prompts.
- Separately check consistency on outputs needing matched pairs like brackets or key-value pairs.
Use block size thinking for long outputs
- Add block diffusion models to candidates for variable-length generation tasks.
- Tune block size by GPU utilization and compare throughput.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the paper's method: draft → full rewrite rounds)

Handle the task below in a diffusion style. Don't write left to right in one pass; make a full draft first, then refine all parts simultaneously over several rounds.

Task: {{document or code description to write}}

1) Round 0: Produce a rough full-length draft, leaving low-confidence parts as [MASK].
2) Rounds 1-3: Each round, look at context on both sides, fill the [MASK]s, and find and fix inconsistencies (mismatched names, brackets, numbers).
3) At the end, give the final version and a summary of what changed in each round.

Next to try: compare against a single-pass version of the same task and count the errors.

Terminology

Autoregressive modelWrites text left to right one token at a time. Like writing in pen: you cannot fix what you already wrote.

Masked diffusion (MDLM)Generates text by repeatedly filling in randomly masked tokens. Basically BERT's fill-in-the-blank used as a generator.

ELBOA lower bound used in place of the hard-to-compute likelihood. A proxy score for comparing models by perplexity.

Block diffusionRuns diffusion block by block, conditioned on earlier blocks. It removes the fixed-length limit.

KV cachingStores already computed keys and values for reuse. Avoids recomputing earlier parts so generation is faster.