Hugging Face Releases "The Ultimate MoE Training Handbook"

Key Info

Hugging Face released "The Ultimate MoE Training Handbook," a free and open resource based on 800k GPU hours of experiments, addressing practical questions training teams face when building Mixture-of-Experts models.

Highlights

• 18 of the 20 best open models today are Mixture-of-Experts, yet training know-how remains concentrated in a few teams.
• The handbook covers expert count and size, shared vs. non-shared experts, all-to-all communication overhead, router choices (softmax, sigmoid, sqrt(softplus)), auxiliary loss vs. DeepSeek's loss-free bias, and more.
• It is organized around real questions training teams hit, such as why a run is 25x slower with no errors in logs.
• The resource is free and open.