DeepSeek Open-Sources DeepInfer, the Inference Engine Behind Its Production Services

What happened

DeepSeek announced the open-source release of its inference framework DeepInfer on X on Oct 20, publishing the code on GitHub under the Apache 2.0 license, SiliconANGLE reported. In an accompanying deployment Q&A, DeepSeek called DeepInfer "the high-performance inference framework that powers DeepSeek's large-scale production services," built to solve what it describes as the core problem of serving models to a massive user base "at a reasonable price."

The first release supports only one model, DeepSeek-V4.1-Flash, loading the updated 1017 snapshot released Oct 17 by default. On hardware it launches with support for both Nvidia SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs; older Nvidia architectures such as Hopper are not on the list. The project's README notes the framework was "originally built primarily on vLLM," with ideas drawn from SGLang and TensorRT-LLM.

DeepSeek also released deepseek-recipe, an MIT-licensed token encoding and decoding layer, MarkTechPost reported. Within hours of going live, the GitHub repository had collected thousands of stars (AIWeekly logged over 5,500 stars and 415 forks). The release landed one day after the U.S. Department of Homeland Security issued its DeepSeek removal order for federal agencies.

Key facts

1. License: DeepInfer is released under Apache 2.0; the companion deepseek-recipe tokenizer layer is MIT-licensed.
2. Dual-chip support: Launches with Nvidia SM100 Blackwell GPUs and Huawei Ascend 950 NPUs; all published benchmarks ran on Nvidia hardware, and two features guaranteeing bit-identical outputs work only on Nvidia (SiliconANGLE).
3. Production figures: Per DeepSeek's own claims: a single cluster serves 155 trillion tokens per day; at off-peak pricing that works out to about $38 in revenue per GPU per hour; input cache hit rate is 96%. All figures are company-reported and unverified by third parties.
4. Core technique: Speculative decoding via DSpark: a draft model proposes five tokens at a time and a confidence head predicts which the main model will accept, spending verification compute only on the likeliest candidates; DeepSeek estimates a 4%–7% throughput gain at production load (SiliconANGLE).
5. First-release limits: Offline inference only; the KV-cache system, prefill/decode disaggregation and load balancing are not yet included (MarkTechPost).