DeepSeek Open-Sources DeepInfer, Its Production Inference Engine — Blackwell and Ascend 950 Only, No KV Cache Layer

This piece is an analysis of public materials: DeepSeek's GitHub repository and deployment notes, the Hugging Face model card, DeepSeek's API pricing page, and press coverage. We did not build the code or reproduce any benchmark.

Who should care

• Teams self-hosting open-weight models who want to see how a frontier lab actually serves a 763B-parameter MoE model at scale.
• Inference and platform engineers working on vLLM or SGLang stacks, since DeepInfer descends from vLLM and documents its production tricks.
• GPU clouds and neoclouds with B200/GB200 capacity, who can now benchmark DeepSeek's own serving path against their current stack.
• U.S. public-sector teams and contractors, who need to separate "open-source code we run ourselves" from "DeepSeek products and services" in their compliance reviews.

What DeepSeek actually released

The DeepInfer repository describes the project as a "high-performance, large-scale inference framework for DeepSeek models" and says this release provides "offline inference with the core features needed for serving at scale." The acknowledgments say it was built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM.

Feature-wise, the README lists attention data parallelism with MoE expert parallelism and load balancing (EP + EPLB), a speculative decoding scheme called DSpark that uses a confidence signal to decide which draft tokens to verify, bounded replay to cut prefill work around sliding-window attention, asynchronous scheduling, and structured output via regex, JSON schema, or EBNF grammars. Batch-invariant inference and deterministic sampling are documented as NVIDIA-only.

The target model's Hugging Face card lists a 763B-parameter model with a recommended 1M-token context, with both repository and weights under MIT. DeepSeek also published deepseek-recipe, which tokenizes requests the same way its official API does, so self-hosted output formatting can match the hosted service.

The numbers DeepSeek disclosed (self-reported)

The most unusual part of the release is a deployment Q&A document that describes DeepSeek's production topology and economics. None of these figures have been independently audited.

The README's Pareto charts were measured on 4 SM100 GPUs for prefill and 64 for decode. In other words, the charts assume a multi-node Blackwell setup, and the economics describe a deployment of at least 128 GPUs, not a single node.

Unconfirmed or forward-looking

• Hopper and older GPUs: The README names only SM100 and Ascend 950. Whether DeepInfer builds or performs well on H100/H200 or A100 is unconfirmed.
• Missing pieces: SiliconANGLE reports DeepSeek said the KV cache and load-balancing code is tightly coupled to internal infrastructure, and that future models' inference code will land in the same repository. No timeline was given.
• Federal context: SiliconANGLE and The Next Web both reported that, a day before the release, the U.S. government ordered federal civilian agencies to remove DeepSeek products. We could not locate the directive's text for this article, so its scope as applied to self-hosted Apache-licensed code is unconfirmed.
• Independent benchmarks: As of publication there are no third-party reproductions of DeepSeek's charts.

What it means in practice

For most teams, DeepInfer is a reference implementation first and a drop-in engine second. Engineers running vLLM or SGLang get a documented look at how DeepSeek handles wide expert parallelism, confidence-gated speculative decoding, and prefill reduction in production. The Next Web reports the vLLM project said it would fold ideas from DeepInfer into its own V4.1-Flash support, so many users may benefit indirectly without switching engines.

What it does not do is make small-scale self-hosting cheap overnight. The disclosed economics depend on 128-GPU deployment units plus a KV cache layer that was not released. A team with a handful of GPUs is still better served by the vLLM and SGLang paths listed on the model card.

For API users, the pricing page lists V4.1-Flash at $0.15–$0.30 per million cache-miss input tokens and $0.60–$1.20 per million output tokens. Peak hours are weekdays 01:00–04:00 and 06:00–10:00 UTC, which works out to roughly 9 p.m.–midnight and 2–6 a.m. Eastern (6–9 p.m. and 11 p.m.–3 a.m. Pacific) while daylight time is in effect. Most of the U.S. workday therefore already falls in the half-price off-peak window.

Limitations

This article relies on DeepSeek's GitHub repository (README, deployment Q&A, license), the Hugging Face model card, DeepSeek's API pricing page, and press reports checked on October 20–21. Throughput, revenue, and performance figures are DeepSeek's own. Support for older GPUs, the scope of the reported federal directive, and third-party benchmark results could not be confirmed. API prices can change.

Last verified 2026-10-21.