DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models

TL;DR

This work tackles bandwidth bottlenecks in distributed foundation-model pretraining by introducing Desynced Low Communication Adaptive Optimizers (DES-LOC), which decouple synchronization frequencies for model parameters and optimizer states. The approach yields provable convergence for SGDM and Adam variants under realistic assumptions, while delivering substantial communication reductions (up to ~$170\times$ vs DDP and ~$2\times$ vs LocalAdam) and robust scalability to billion-scale language models. Empirically, DES-LOC shows predictable momentum-state dynamics governed by half-lives, enabling practical configurations that maintain perplexity while improving wall-clock efficiency, even under system failures or heterogeneous data. The results suggest DES-LOC as a scalable, fault-tolerant alternative to DDP for distributed foundation-model training, with clear guidelines to tune parameter and momentum synchronization frequencies.

Abstract

Scaling foundation model training with Distributed Data Parallel (DDP) methods is bandwidth-limited. Existing infrequent communication methods like Local SGD were designed to synchronize only model parameters and cannot be trivially applied to adaptive optimizers due to additional optimizer states. Current approaches extending Local SGD either lack convergence guarantees or require synchronizing all optimizer states, tripling communication costs. We propose Desynced Low Communication Adaptive Optimizers (DES-LOC), a family of optimizers assigning independent synchronization periods to parameters and momenta, enabling lower communication costs while preserving convergence. Through extensive experiments on language models of up to 1.7B, we show that DES-LOC can communicate 170x less than DDP and 2x less than the previous state-of-the-art Local ADAM. Furthermore, unlike previous heuristic approaches, DES-LOC is suited for practical training scenarios prone to system failures. DES-LOC offers a scalable, bandwidth-efficient, and fault-tolerant solution for foundation model training.

Figures (16)

• Figure 1: We present a toy problem where DES-LOC ($K_x=192,K_u=192,K_v=692$) and LocalAdam ($K=K_x$) both converge to the optimum (overlapping in \ref{['fig:toy_distances_iid']}). Methods keeping optimizer states local DiLoCoPhoton fail, causing oscillations without convergence. Periodically resetting states LLMFLDEPT similarly stalls due to repeated oscillations. We optimize the non-convex function $f(x_1,x_2)=(1 - x_1)^2+100(x_2 - x_1^2)^2$ with $M=256$ workers and IID Gaussian noise ($\sigma=1.5$).
• Figure 2: Relative rates of change for first and second momenta across rounds using standard LocalADOPT/Adam ($K=64$). For ADOPT ($\beta_2=0.9999$), increasing $\beta_1\geq0.99$ greatly slows the first-momentum rate of change. The second momentum evolves $\sim100\times$ slower (note y-axis is in log scale), consistent with their decay rates and half-lives. For Adam, higher $\beta_1,\beta_2$ slow both momenta.
• Figure 3: Model perplexity for DES-LOC (ADOPT, $\beta_1=0.95,\beta_2=0.9999$), varying synchronization periods independently (others fixed at $K_b$). Parameter synchronization (a) is critical, with sharp degradation at higher periods. Second-momentum synchronization (b) minimally affects performance due to its large half-life ($\tau_{0.5}(\beta_2)\gg K_b$). First-momentum synchronization significantly improves perplexity (c) only when the baseline matches its half-life ($K_b=16$), having minimal impact otherwise (d). Parameters and second momentum behave similarly across sync frequencies (\ref{['app:additional_results']})
• Figure 4: Setting $K_x=K$, $K_u=3K_x$, and $K_v=6K_x$, DES-LOC achieves a $\mathbf{2}\times$ communication reduction over LocalAdam, matching performance at high (a) and low (b) frequencies for LocalAdam and heuristic baselines (see \ref{['sec:exp_setup']}). We demonstrate robustness to the addition of new workers by doubling worker count at step $1536$ (c,d); DES-LOC and LocalAdam remain stable in perplexity/gradient norms, outperforming heuristic methods and ad-hoc optimizer-state averaging.
• Figure 5: DES-LOC matches LocalAdam perplexity for billion-scale model training at half the communication cost ($K_x=256,K_u=3K_x,K_v=6K_x$), representing a $\mathbf{170}\times$ reduction over DDP. Though initially behind DDP, both DES-LOC and LocalAdam quickly converge to competitive perplexity at longer training horizons. Federated Averaging (keeping optimizer states) achieves reasonable performance (a) but suffers activation growth (b) and parameter-norm growth (\ref{['app:additional_results']}), potentially due to noisy local updates, raising concerns for extended training ($\geq11$ trillion tokens SmolLM2).

Theorems & Definitions (10)

• Theorem 1
• Theorem 2: Informal
• Lemma 3
• proof
• Lemma 4
• proof
• Lemma 5
• proof
• Theorem 6: Full version of Theorem 2
• proof