CoreAuto's Rohan Anil reports training a sparse MoE across two clusters half a world apart over a 25 Gbps link
The clusters differed in size and GPU type; the post also reports that butterfly mixing, which swaps all-reduce for pairwise exchanges between eight learners, matched standard training.
Why it matters
Frontier training normally assumes one large co-located cluster, but power, grid equipment and land are the main limits on new data centres, so much GPU capacity sits in scattered smaller sites. This position post argues that learning algorithms should be designed to use those islands of compute together, and reports a working run across two very different clusters. It is a blog post from a startup, not a peer-reviewed paper. Watch whether the approach scales beyond two sites and to larger models.
Key facts
- Sparse mixture-of-experts model trained across two clusters
- Clusters differed in size and GPU type
- Sites nearly halfway around the world from each other
- Bidirectional link of 25 Gbps
- Butterfly mixing with pairwise exchanges between eight learners matched standard training