Hugging Face publishes Ultimate MoE Training Handbook from ~500k GPU-hours
Nouamane Tazi’s team documents Megatron-LM MoE settings, serving and RL from Alps GH200 runs; Space ships 50+ interactive figures.
Why it matters
Mixture-of-Experts models now lead many open rankings, but public training recipes stay scarce. Hugging Face’s handbook turns roughly half a million GPU-hours on CSCS’s Alps cluster into a Megatron-LM runbook on expert counts, routers, communication and RL. It ships scripts, recommended flags and more than fifty interactive figures rather than marketing copy. Watch whether labs adopt the flags in public runs and whether the serving section changes how mid-size MoEs are deployed.
Key facts
- ~500k GPU-hours on Alps GH200 (CSCS)
- Megatron-LM MoE runbook with scripts and flags
- 50+ interactive figures on Hugging Face Space
- Covers profiling, cluster training, serving and RL
- Announced by @Nouamanetazi 2026-11-05 18:39Z