Baidu open-sources ERNIE 5.5 Flash Base, a 280B-A14B pre-training MoE checkpoint
Trained from scratch on 33.5T tokens with up to 128K context (1M for fine-tune/infer); Baidu reports MMLU 93.3 among peer base models, with FP8 and BF16 weights on Hugging Face.
Why it matters
Releasing the end-of-pretraining checkpoint, not a chat-tuned model, lets labs and startups run their own mid-training and RL instead of starting from an instruction model. The 280B-A14B sparse MoE with hybrid compressed attention is aimed at cheap further training and long context, and Baidu's peer-base table puts it ahead of DeepSeek-V4-Flash-Base and others on MMLU and several code tests. Licence limits and whether community fine-tunes close the gap to post-trained ERNIE 5.5 Flash are the next checks.
Key facts
- 280B total parameters, 14B activated per token
- Pre-trained from scratch on 33.5T tokens
- Training context up to 128K; fine-tune/inference up to 1M
- End-of-pretraining weights, no mid- or post-training
- Baidu reports MMLU 93.3 vs peer base models
- FP8 and BF16 safetensors on Hugging Face and ModelScope