DeepInfer: High-performance inference framework for DeepSeek models

Who Should Read

Infra or backend developers self-hosting DeepSeek models on their own GPUs who care about inference cost and throughput.

Core Mechanics

• Engine behind DeepSeek's large-scale service
  • DeepInfer is the framework that powers DeepSeek's large-scale production services.
  • This release covers offline inference and core features, under Apache-2.0.
• Narrow model and hardware support
  • The only supported model is DeepSeek-V4.1-Flash.
  • Hardware is NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
• Plenty of speed features
  • Draft tokens speed up decoding, and confidence estimates schedule verification (DSpark).
  • Asynchronous scheduling overlaps CPU batch scheduling with device execution and is on by default.
  • Parallelism includes Attention DP / balanced CP and MoE EP + EPLB.
• Reproducibility options
  • DEEPINFER_BATCH_INVARIANT=1 keeps results identical across batch compositions (NVIDIA only).
  • Deterministic sampling gives identical outputs for identical input tokens and sampling parameters.
• Structured outputs built in
  • Generation can be constrained by a regex, JSON schema, or EBNF grammar.
  • Pass so_args={"schema_type": ..., "schema": ...} to SamplingParams.
• Not a full serving stack yet
  • The KV cache system is missing from this release.
  • VE/Prefill/Decode disaggregation and load balancing across data-parallel replicas are also missing.
• Demanding build requirements
  • You need Linux, GCC 13+, and Python 3.12.
  • NVIDIA needs CUDA 13, and Ascend needs CANN 9.2.

Community Reactions

• Throughput and revenue figures cited
  • One commenter quoted a Zhihu post saying single-cluster throughput is 155 trillion tokens/day.
  • The same post reportedly gives revenue of $38 per card per hour at off-peak pricing.
• Doubts about comparing the $38 figure
  • A commenter noted H100 rentals run about $2.75 per hour.
  • They pointed out the card type was not specified.
• Missing KV cache system flagged
  • Several comments noted the KV cache system is absent.
  • One said the Zhihu post lists retaining KV cache for 72+ hours as a goal.
• Curiosity about gains over vLLM and SGLang
  • Since the repo says it was inspired by both, a commenter wondered how big the performance gain is.
  • Another hoped it might improve tool calling, which they said third-party DeepSeek providers handle worse.
• Mixed views on Ascend support
  • Some argued Huawei Ascend support could challenge Nvidia's hold.
  • Others countered that regulation keeps Huawei hardware out of the US and EU, so Nvidia stays.
• Hands-on experience with V4.1 Flash
  • One commenter said they switched from a Claude Max subscription to the DeepSeek API and found V4.1 Flash fast with almost no refusals.

How to Apply

• Check the README requirements today
  • Confirm whether your GPUs are SM100 (NVIDIA) or Ascend 950.
  • Check that you meet Python 3.12, GCC 13+, and CUDA 13.
• Use deterministic options in regression tests
  • If tests need identical outputs, set DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0.
  • Pass a fixed seq_seed in SamplingParams (NVIDIA only).
• Use structured outputs to force JSON
  • If your pipeline often fails parsing, pass schema_type and schema via so_args.
  • Run the examples folder first to check the format.
• List what you must build before self-hosting
  • Write down whether you need to build the KV cache system and load balancing yourself.
  • Benchmark against your current vLLM or SGLang setup under the same conditions.

Try It

# Run in a shell (Linux, empty Python 3.12 env, NVIDIA SM100 + CUDA 13 required)

git clone https://github.com/deepseek-ai/DeepInfer
cd DeepInfer

# Use ascend instead for Ascend; keep this exported where you run DeepInfer
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

# On first run, deepseek-ai/DeepSeek-V4.1-Flash-1017 downloads into the Hugging Face cache
# To use a local copy: export DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017
python examples/offline_inference.py

# Next: deterministic sampling
# DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0 python examples/offline_inference.py
# (pass a fixed seq_seed in SamplingParams)

Terminology

Related Papers

• Mercury 3: Diffusion at scale

  Scaling diffusion LLM 40x gives Mercury 3: 4–8x faster than speed-tier models, similar quality

• DeepSeek banned from U.S. federal civilian agencies under new DHS order

  DHS order bans DeepSeek from U.S. federal civilian agency IT systems

• Timestral: Mistral's time-series foundation models

  A 209M time-series model ranks first on three benchmarks and runs on CPU.

• DeepSeek V4.1 Flash 1017 (Max)

  DeepSeek V4.1 Flash 1017 Max scores 45 on the Intelligence Index at 211 tok/s and $0.30 input.

• If you don't run your own LLM proxy, how do you know whats going on?

  Running your own LLM proxy shows which keys, models, and cache misses drive your usage.

• DeepSeek: DeepSeek v4.1 Flash 1017

  DeepSeek V4.1 Flash 1017: $0.05/M input, 1M context, built for agents

• DeepSeek-v4.1-Flash-1017

  DeepSeek V4.1 Flash minor update lifts coding and agent benchmarks while cutting run time and output length

• Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware

  Pick hardware and a model to preview local LLM speed before buying

Related Resources

• https://github.com/deepseek-ai/DeepInfer
• https://zhuanlan.zhihu.com/p/2095560295930508953
• https://mmoustafa.com/blog/so-you-want-to-use-openrouter/