Accelerating Decentralized Optimization via Overlapping Local Steps

TL;DR

The paper tackles communication bottlenecks in decentralized optimization by introducing Overlapping Local DSGD (OLDSGD), which overlaps computation and communication while using a CTA-based update in a fully decentralized network. OLDSGD preserves an SGD-like average update for the global model, ensuring the same iteration complexity as Local DSGD in nonconvex settings and achieving improved wall-clock convergence. Theoretical results establish nonasymptotic convergence rates with explicit bounds, and extensive experiments on CIFAR-10 and GPT-2 finetuning demonstrate substantial speedups, particularly under higher communication delays and with larger local steps. The method requires minimal changes to existing frameworks and scales effectively on ring topologies, making it practical for edge and federated decentralized learning scenarios.

Abstract

Decentralized optimization has emerged as a critical paradigm for distributed learning, enabling scalable training while preserving data privacy through peer-to-peer collaboration. However, existing methods often suffer from communication bottlenecks due to frequent synchronization between nodes. We present Overlapping Local Decentralized SGD (OLDSGD), a novel approach to accelerate decentralized training by computation-communication overlapping, significantly reducing network idle time. With a deliberately designed update, OLDSGD preserves the same average update as Local SGD while avoiding communication-induced stalls. Theoretically, we establish non-asymptotic convergence rates for smooth non-convex objectives, showing that OLDSGD retains the same iteration complexity as standard Local Decentralized SGD while improving per-iteration runtime. Empirical results demonstrate OLDSGD's consistent improvements in wall-clock time convergence under different levels of communication delays. With minimal modifications to existing frameworks, OLDSGD offers a practical solution for faster decentralized learning without sacrificing theoretical guarantees.

Figures (8)

• Figure 1: Communication-computation schematics of DSGD, LDSGD, and OLDSGD. CU: consensus update, LU: local update, Comm.: communicaiton.
• Figure 2: Convergence w.r.t. time under different tasks (c=1). The first row presents all homogeneous cases, while the second row presents all heterogeneous cases.
• Figure 3: Convergence w.r.t. time under different tasks (c=5). The first row presents all homogeneous cases, the second presents all heterogeneous cases.
• Figure 4: Scalability of OLDSGD on VGG11 (a) and ResNet18 (b) (CIFAR-10, ring topology, $\tau=5$). Near-linear speedup is achieved up to 16 agents (14× for VGG11, 13× for ResNet18), with diminishing returns beyond due to ring topology constraints. Loss curves demonstrate accelerated convergence with increasing parallelism.
• Figure 5: Loss curves of ResNet18 training w.r.t. iteration under a low data heterogeneity level. LED/OLED diverges in all cases and OLGT performs worse than LUGT under local steps.

Theorems & Definitions (15)

• Theorem 3.6
• Corollary 3.7
• Remark 3.8
• Lemma C.1
• proof
• Lemma C.2
• proof
• Lemma C.3
• proof
• Lemma C.4