Ordered Local Momentum for Asynchronous Distributed Learning under Arbitrary Delays

TL;DR

This work introduces OrLoMo, the first method for asynchronous distributed learning with local SGD updates and momentum, by aggregating local momentum in global-iteration order to form the server-side momentum. It provides a non-convex convergence guarantee under arbitrary delays, without requiring delay bounds or bounded gradients, and characterizes the gradient-iteration trade-offs with a concrete rate. Empirically, OrLoMo outperforms both synchronous and other asynchronous baselines, showing robustness to heterogeneity and large delays across multiple datasets and architectures. The approach reduces communication without sacrificing convergence or speed, making it practically impactful for large-scale distributed training. Note: All mathematical expressions are presented with explicit dollar-delimited formatting as required.

Abstract

Momentum SGD (MSGD) serves as a foundational optimizer in training deep models due to momentum's key role in accelerating convergence and enhancing generalization. Meanwhile, asynchronous distributed learning is crucial for training large-scale deep models, especially when the computing capabilities of the workers in the cluster are heterogeneous. To reduce communication frequency, local updates are widely adopted in distributed learning. However, how to implement asynchronous distributed MSGD with local updates remains unexplored. To solve this problem, we propose a novel method, called \underline{or}dered \underline{lo}cal \underline{mo}mentum (OrLoMo), for asynchronous distributed learning. In OrLoMo, each worker runs MSGD locally. Then the local momentum from each worker will be aggregated by the server in order based on its global iteration index. To the best of our knowledge, OrLoMo is the first method to implement asynchronous distributed MSGD with local updates. We prove the convergence of OrLoMo for non-convex problems under arbitrary delays. Experiments validate that OrLoMo can outperform its synchronous counterpart and other asynchronous methods.

Figures (6)

• Figure 1: Update demonstration when $t=8, K=3$.
• Figure 2: Test accuracy curves of SqueezeNet model when $K=16, S=16$.
• Figure 3: Test accuracy curves of ResNet20 model when $K=16, S=16$.
• Figure 4: Training loss curves of ResNet20 model when $K=16, S=16$.
• Figure 5: Training loss curves of SqueezeNet model when $K=16, S=16$.

Theorems & Definitions (26)

• Remark 1
• Remark 2
• Lemma 1
• Lemma 2
• Lemma 3
• Lemma 4
• Theorem 1
• Remark 3
• Remark 4
• Remark 5