DeepSeek Open-Sources DeepInfer Inference Framework for Nvidia and Huawei Ascend Chips

Full report

DeepSeek describes DeepInfer as "the high-performance inference framework that powers DeepSeek's large-scale production services." The first release supports a single model, DeepSeek-V4.1-Flash, on two kinds of datacenter hardware: Nvidia SM100 GPUs (CUDA 13) and Huawei Ascend 950 NPUs (CANN 9.2).

Technically it includes custom operators (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, TileLang operators), MoE expert parallelism with EPLB load balancing, DSpark speculative decoding (default draft length 5), asynchronous scheduling, batch-invariant inference and deterministic sampling (Nvidia only), and structured outputs (regex / JSON schema / EBNF). The README credits it as "originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM."

This release excludes the KV cache system, prefill/decode disaggregation and cross-replica load balancing; benchmarks were measured on 4 SM100 GPUs (prefill) and 64 (decode). DeepSeek also open-sourced deepseek-recipe, the tokenizer/encoding component matching its official API.

Context: media reported that DeepSeek said in April 2025 it did not plan to open-source its full inference engine, making this a reversal; the announcement came a day after the US federal government ordered federal agencies to ban DeepSeek. DeepSeek claims a single V4.1-Flash cluster serves 155 trillion tokens per day (not independently verified).