Key Info

DeepSeek has open-sourced DeepInfer, a high-performance large-scale inference framework, under the Apache-2.0 license.

Highlights

  • Includes optimized kernels plus model and sampler runtime
  • Parallelism support: Attention DP / balanced CP, and MoE EP with EPLB
  • Features DSpark and confidence-scheduled verification
  • Adds SWA bounded replay and an asynchronous zero-overhead batch scheduler
  • Supports batch-invariant inference, deterministic sampling, and structured output