StepFun Releases Step 5 Preview Open Weights: 600B Sparse MoE with 1M-Token Context

Key Info

StepFun released Step 5 Preview open weights in BF16 and FP8 on Hugging Face and ModelScope, alongside a technical blog and an updated open-source Step Code agent harness (v0.1.4).

Highlights

• Sparse MoE model: 600B total parameters, 27B active per token
• 1M-token context window with native text and vision input
• Targeted at long-horizon agentic work: coding, software engineering, professional knowledge work and finance
• Technical blog details a 92-layer narrow-and-deep architecture for implicit multi-hop reasoning, sparse GQA with 8-token block indexing cutting indexing and top-k overhead to 1/8, FP32 precision on key states with fine-grained gradient clipping, million-scale verifiable environments, and RL for long-horizon tasks with context compression in policy training
• Step Code v0.1.4 has had four releases since open-sourcing, roughly one every 3 working days, mostly driven by community feedback