SeedFlood: A Step Toward Scalable Decentralized Training of LLMs

TL;DR

This work presents a new approach to decentralized training-SeedFlood-designed to scale for large models across complex network topologies and achieve global consensus with minimal communication overhead, and consistently outperforms gossip-based baselines in both generalization performance and communication efficiency.

Abstract

This work presents a new approach to decentralized training-SeedFlood-designed to scale for large models across complex network topologies and achieve global consensus with minimal communication overhead. Traditional gossip-based methods suffer from message communication costs that grow with model size, while information decay over network hops renders global consensus inefficient. SeedFlood departs from these practices by exploiting the seed-reconstructible structure of zeroth-order updates and effectively making the messages near-zero in size, allowing them to be flooded to every client in the network. This mechanism makes communication overhead negligible and independent of model size, removing the primary scalability bottleneck in decentralized training. Consequently, SeedFlood enables training in regimes previously considered impractical, such as billion-parameter models distributed across hundreds of clients. Our experiments on decentralized LLM fine-tuning demonstrate thatSeedFlood consistently outperforms gossip-based baselines in both generalization performance and communication efficiency, and even achieves results comparable to first-order methods in large scale settings.

Figures (7)

• Figure 1: Task performance vs. Total communication cost plot of different decentralized training methods. SeedFlood ($\bigstar$) is extremely efficient---with $10^{2}$–$10^{7}\times$ less communication bytes--while maintaining a reasonable performance level compared to its rivals and strong-but-impractical baseline methods.
• Figure 2: Consensus dynamics of a single gradient under gossip-based model averaging (a) and flooding-based gradient dissemination (b). In gossip, time-varying gradient coefficients induce prohibitive aggregation cost. In contrast, flooding propagates each gradient with a fixed coefficient, without gradual diffusion, enabling uniform and fast application across clients and leading to perfect consensus.
• Figure 3: Task performance vs. Communication cost plots on diverse tasks in the SuperGLUE benchmark. SeedFlood consumes near-zero communication cost (Kilobytes), and is spotted in the left of all panels. SeedFlood also records reasonably high task performances; it outperforms existing communication-efficient rival baselines such as ChocoSGD or LoRA variants, while stays competitive to first-order methods such as DSGD. While the absolute performance levels seem to vary slightly between different tasks, SeedFlood consistently remains as a strong alternative in this trade-off.
• Figure 4: Relative GMP differences (percentage points) from the 16-client baseline as the network size increases, comparing ring (left) and meshgrid (right) topologies.
• Figure 5: Runtime measurements for applying multiple zeroth-order gradient messages on OPT-2.7B using a single A100 GPU. SubCGE is several orders of magnitude faster than MeZO.