DiffusionGemma: 4x Faster Text Generation
TL;DR Highlight
Generating 256 tokens in parallel makes DiffusionGemma up to 4x faster on local GPUs
Who Should Read
A backend or app developer building low-latency features like inline editing or rapid iteration on local GPUs
Core Mechanics
- It writes 256 tokens at once, not one by one
- DiffusionGemma is a 26B MoE open model under Apache 2.0.
- Unlike autoregressive LLMs, which predict tokens left to right, it generates a whole text block simultaneously.
- Speed is 1000+ tok/s on an H100
- The post claims up to 4x faster token output on dedicated GPUs.
- It reports 1000+ tokens per second on a single NVIDIA H100 and 700+ on a GeForce RTX 5090.
- It is fast only for local, low-concurrency use
- A local single-user GPU sits idle waiting for the next token, so parallel decoding helps.
- In high-QPS cloud serving the gains shrink and serving costs can be higher.
- Unified-memory machines like Apple Silicon may not see the same speedup.
- Quality is lower than standard Gemma 4
- It prioritizes speed and parallel layout generation, so overall output quality is lower.
- Google recommends standard Gemma 4 when maximum quality is needed.
- Bi-directional attention suits editing and infilling
- All 256 tokens attend to each other, which helps inline editing, code infilling, and non-linear structures.
- The model reviews the whole block and fixes mistakes as it goes.
- It starts from noise and refines over passes
- It begins with a canvas of random placeholder tokens.
- Each pass locks in correct tokens and uses them as context for the rest.
- It fits in 18GB of memory
- Only 3.8B of the 26B parameters are active during inference.
- Quantized, it fits in 18GB VRAM on high-end consumer GPUs.
Community Reactions
- Fast diffusion felt like pair programming, one user says
- They used Mercury (a diffusion model) in OpenCode and liked it for being stupid fast, not smart.
- It felt less like a prompt-wait-hope slot machine and made them use small models more.
- Commenter says it relieves the edge memory-bandwidth bottleneck
- Consumer GDDR/LPDDR has lower bandwidth than HBM, and serial requests can't be batched to share weights.
- Computing tokens in parallel eases that bottleneck, they argue.
- One user got far below advertised speed on a 3090 Ti
- They ran a Q4 quant on patched llama.cpp and saw nowhere near the claimed speeds.
- Watching answers fill in was fun, and the SVG pelican test came out minimal but on brief.
- Skeptic says local model quality isn't worth trading for speed
- They find local models noticeably worse than even the cheapest APIs and won't give up any quality.
- They ask which production use cases people actually plan.
- Commenter questions latency for very short replies
- If the reply is one word, the model still processes a 256-token block, so it may be slower to first token.
- Commenter notes it needs a llama.cpp fork for now
- Unsloth has published the model on HF, but running it currently requires a llama.cpp fork.
How to Apply
- Download the weights and time it locally with Transformers
- On a 24GB-class GPU, pull the weights from Hugging Face and run them with Transformers.
- Compare tok/s and quality side by side against standard Gemma 4 on the same prompts.
- Trial it on inline editing or code-infilling features
- Pick latency-sensitive features like editor auto-fix, code infilling, or quick iteration.
- Use it for drafts and suggestions rather than final output, given the lower quality.
- Keep standard models for high-concurrency servers
- Leave cloud serving with many requests on a standard autoregressive model.
- On Apple Silicon, measure first since the speedup may not appear.
- Fine-tune it for narrow tasks with Unsloth or NeMo
- For a narrow, structured-output task, follow the Unsloth or NVIDIA NeMo fine-tuning tutorials.
- The post uses Sudoku as its fine-tuning example.
Try It
# Paste into Claude or ChatGPT (prompt for judging whether to adopt DiffusionGemma)
Help me decide whether to adopt a local diffusion text model (DiffusionGemma: 26B MoE, 256-token parallel generation, lower quality than standard Gemma 4, fast only in local low-concurrency use) for our service.
[Our feature]
{{feature description}}
[Runtime environment]
{{GPU type, concurrent requests}}
[Quality requirement]
{{e.g. drafts are fine / final output}}
Summarize in a table:
1. Is this feature low-latency and low-concurrency?
2. Does it need bi-directional context like inline editing or infilling?
3. Can we tolerate the quality drop?
4. Recommendation (yes/no) with 3 lines of reasoning
5. Three metrics to measure in a benchmarkTerminology
Related Papers
DeepInfer: High-performance inference framework for DeepSeek models
DeepSeek open-sourced DeepInfer, its production inference engine, under Apache-2.0.
Mercury 3: Diffusion at scale
Scaling diffusion LLM 40x gives Mercury 3: 4–8x faster than speed-tier models, similar quality
DeepSeek banned from U.S. federal civilian agencies under new DHS order
DHS order bans DeepSeek from U.S. federal civilian agency IT systems
Timestral: Mistral's time-series foundation models
A 209M time-series model ranks first on three benchmarks and runs on CPU.
DeepSeek V4.1 Flash 1017 (Max)
DeepSeek V4.1 Flash 1017 Max scores 45 on the Intelligence Index at 211 tok/s and $0.30 input.
If you don't run your own LLM proxy, how do you know whats going on?
Running your own LLM proxy shows which keys, models, and cache misses drive your usage.
DeepSeek: DeepSeek v4.1 Flash 1017
DeepSeek V4.1 Flash 1017: $0.05/M input, 1M context, built for agents
DeepSeek-v4.1-Flash-1017
DeepSeek V4.1 Flash minor update lifts coding and agent benchmarks while cutting run time and output length