DeepSeek-v4.1 Flash: Pushing the Limits of KV Cache Compression

TL;DR Highlight

DeepSeek-V4.1 Flash cuts KV cache to 1/4 runtime and 1/8 persistent storage versus V4 Flash.

Who Should Read

Backend or LLM infra developers struggling with prefill cost and KV cache storage in long-context agent workloads

Core Mechanics

The goal is KV cache compression
- Long-horizon agents keep growing context, and tool calls create heavy prefill load.
- Storage pressure on HBM and external SSD is described as a barrier to scaling.
1/4 and 1/8 of V4 Flash
- At the same sequence length, runtime KV cache is about 1/4 of DeepSeek-V4-Flash.
- Persistent KV cache is about 1/8, and overall performance is reported as better than V4-Flash.
Only half the layers for prefill
- Of 40 layers, only the first 20 (Encoder) run for prefill, and the last 20 (Decoder) share KV projected from the Encoder's final states.
- Activated parameters are 8B for prefill and 16B for decode (552B total, up to 1M context).
Compressed along three axes
- Channel: one 512-dim latent shares keys and values across all attention heads.
- Sequence: the Encoder merges 2 adjacent positions into 1 cache entry.
- Layer: only 3 Encoder caches and 1 Decoder cache are kept.

Community Reactions

Held up in very long sessions
- One user ran an automatic incremental context compactor with DS41f, with the model at 300k-400k and an effective session length of about 5M.
- They report speed and quality held up.
Preview speed experience
- The same user tried the 400tok/s preview and called it nuts.
- They let it compact repeatedly for a day on tough problems.
Prefix caching impact
- One commenter said KV cache compression plus prefix caching makes queries converge on practically free.

How to Apply

Measure your agent's prefill share
- Aggregate input-to-output token ratio and cache miss rate per request from today's logs.
- If input dominates, judge whether an 8B prefill / 16B decode design would benefit you.
Convert KV storage cost to bytes per token
- Multiply your model's per-token KV size by the prefix retention length to estimate SSD and host memory needs.
- Compare against the reported ~890 bytes per token to see if storage is a bottleneck.
Add a context compactor to long agent sessions
- Turn on automatic incremental compaction for long-running agents and compare quality after compression.
- Keep the system prompt and tool definitions fixed at the front so the prefix stays stable.

Try It

copy & run

# Simplified version of the paper's idea: paste into Claude or ChatGPT to diagnose your KV cache bottleneck

You are an LLM serving infrastructure engineer. Diagnose the KV cache bottleneck from the info below.
- Average input tokens: {{input tokens}}
- Average output tokens: {{output tokens}}
- Tool calls per agent task: {{tool calls}}
- Prefix cache hit rate: {{hit rate}}
- Model and per-token KV size: {{model}}

Questions: (1) Which is the bigger bottleneck, prefill cost or KV storage cost? (2) Among layer sharing, sequence compression, and low precision (FP4), where should I invest first? (3) Give me 3 measurements to take tomorrow.

Next step: compute the suggested measurements from your real logs, paste them back, and ask for a re-diagnosis.

Terminology

KV CacheStored key/value vectors so attention need not recompute past tokens. The longer the context, the bigger this notepad gets.

prefill / decodePrefill reads the whole prompt at once; decode generates the answer one token at a time. Like reading a book first, then writing a summary.

CED (Causal Encoder-Decoder)Early layers (Encoder) produce KV that later layers (Decoder) share. Early layers take notes and later layers only read them.

SWA (Sliding Window Attention)Attention where each token sees only a recent window (128 here). Skimming recent notes instead of the whole conversation.

Speculative decodingA small draft module proposes several tokens and the main model verifies them in one pass to speed things up. DSpark plays this role here.

FP4A 4-bit number format. Fits the same data in less space, cutting storage and transfer cost.