Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch
Arthur Douillard,
Yanislav Donchev,
Keith Rush,
Satyen Kale,
Zachary Charles,
Zachary Garrett,
Gabriel Teston,
Dave Lacey,
Ross McIlroy,
Jiajun Shen,
Alexandre Ramé,
Arthur Szlam,
Marc'Aurelio Ranzato,
Paul Barham