DeepSeek on Tuesday, October 20, open-sourced DeepInfer, the inference framework that runs its own production services. The code is released under the Apache 2.0 license. DeepSeek also published deepseek-recipe, an MIT-licensed layer that handles token encoding and decoding the way its official API does, so self-hosted setups behave the same way.

What DeepInfer is

Inference is the stage where a trained model answers requests. DeepInfer is the serving software that lets DeepSeek-V4.1-Flash, the company’s fast, low-priced model, handle traffic at scale. It was originally built on vLLM, with ideas from SGLang and TensorRT-LLM, and adds DeepSeek’s own kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels. If you want the basics first, our explainer covers how LLM inference serving works with vLLM.

The first release is narrow. It supports one model, DeepSeek-V4.1-Flash, whose model card lists a 1 million-token context window. It targets Nvidia SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs.

How it works

According to MarkTechPost’s reading of the repository and README, the design stacks five ideas. First, parallelism: attention layers use data parallelism, while mixture-of-experts layers use expert parallelism with a load balancer. Decoding runs at expert-parallel widths of 64 to 128 inside a supernode, while prefill uses the smallest width memory allows.

Second is DSpark, a form of speculative decoding. A small draft model proposes five tokens per step, a “confidence head” predicts how likely each is to be accepted, and the scheduler verifies only the most valuable prefix. DeepSeek says this cuts verified draft tokens by about 30% while accepted length barely changes (3.29 versus 3.20).

Third, on a cache hit, the engine recomputes only the last 128 tokens of sliding-window attention state, which keeps recomputation cost constant regardless of context length and, DeepSeek says, nearly halves prefill compute in the decoder layers. Fourth, CPU scheduling overlaps with GPU work so the chip does not idle. Fifth, it offers deterministic, batch-invariant inference on Nvidia only, plus structured output through regex, JSON schema and EBNF grammars.

What the numbers say

The figures are DeepSeek’s own and have not been independently checked. The company says a single cluster serves 155 trillion tokens per day and reports revenue of $38 per GPU-hour at valley pricing. Its best published benchmark exceeds 25,000 output tokens per second per GPU at 256 requests per rank with 131,072 tokens of context, on a setup of 4 GPUs for prefill and 64 for decode.

Limits of the release

The release covers offline inference only. The KV cache system, prefill/decode disaggregation and load balancing that a full production service needs are not included. Readers hoping to replicate DeepSeek’s exact economics will have to build those pieces themselves. The Decoder reports that code for future DeepSeek models will also ship as open source.

Why it matters

DeepSeek says the aim is to help the community lower compute costs and energy use. The Decoder notes the release adds pricing pressure on Western providers. The timing is also notable: the U.S. recently ordered federal civilian agencies to remove DeepSeek products, as we reported in US bans DeepSeek from federal civilian agencies. Open code is not subject to that ban in private use, but organizations with government customers should check their own rules. Cost comparisons with rivals are covered in Mercury 3 vs MiniMax M3.1 Flash vs DeepSeek V4.1 Flash.

What to watch next

  • Community ports of DeepInfer to other GPUs and to models beyond V4.1-Flash.
  • Whether DeepSeek adds the KV cache and load-balancing layers later.
  • Independent throughput tests on Blackwell and Ascend hardware.
  • Responses from vLLM and SGLang maintainers, who may adopt some techniques.
  • Whether Western providers cut prices in response.

Key takeaways

  • DeepInfer is Apache 2.0 and runs DeepSeek’s production serving for V4.1-Flash.
  • It supports Nvidia Blackwell GPUs and Huawei Ascend 950 NPUs.
  • Techniques include confidence-scheduled speculative decoding and bounded cache replay.
  • The first release is offline inference only and supports a single model.
  • All performance claims are DeepSeek’s own.

Frequently asked questions

Can I run DeepSeek models on my own servers with it?

Yes for V4.1-Flash, if you have supported hardware, but expect to build your own cache and load-balancing layers.

DeepSeek says DeepInfer was originally built primarily on vLLM, with ideas from SGLang and TensorRT-LLM.

Does the federal ban apply to the code?

The ban we reported covers use by federal civilian agencies. Check your own contracts if you serve government customers.