Provably Convergent Decentralized Optimization over Directed Graphs under Generalized Smoothness

TL;DR

This paper tackles decentralized optimization on directed graphs under generalized $(L_0,L_1)$-smoothness, where gradient variation can be rapid and gradient dissimilarity across agents is unbounded. It introduces Clipped-Gradient Tracking (CGT), which clips the gradient-tracking estimates rather than local gradients to stabilize updates and enable convergence without the usual bounded dissimilarity assumption. The authors prove convergence to an $oldsymbol{ar{x}}$ that is $oldsymbol{ abla}F$-stationary in $ ilde{oldsymbol{O}}(1/oldsymbol{}^2)$ iterations, and provide detailed bounds on consensus and gradient-tracking errors under directed communication. Numerical experiments on LIBSVM and CIFAR-10 demonstrate improved stability and faster convergence compared with standard gradient-tracking and clipped-DGD baselines, validating the practical impact of the approach in heterogeneous, large-scale settings.

Abstract

Decentralized optimization has become a fundamental tool for large-scale learning systems; however, most existing methods rely on the classical Lipschitz smoothness assumption, which is often violated in problems with rapidly varying gradients. Motivated by this limitation, we study decentralized optimization under the generalized $(L_0, L_1)$-smoothness framework, in which the Hessian norm is allowed to grow linearly with the gradient norm, thereby accommodating rapidly varying gradients beyond classical Lipschitz smoothness. We integrate gradient-tracking techniques with gradient clipping and carefully design the clipping threshold to ensure accurate convergence over directed communication graphs under generalized smoothness. In contrast to existing distributed optimization results under generalized smoothness that require a bounded gradient dissimilarity assumption, our results remain valid even when the gradient dissimilarity is unbounded, making the proposed framework more applicable to realistic heterogeneous data environments. We validate our approach via numerical experiments on standard benchmark datasets, including LIBSVM and CIFAR-10, using regularized logistic regression and convolutional neural networks, demonstrating superior stability and faster convergence over existing methods.

Figures (5)

• Figure 1: The directed communication graphs used in the evaluation.
• Figure 2: Comparison of loss and gradient norm between Algorithm \ref{['alg:1']} and the gradient tracking algorithm in xin2018linear on the a9a dataset. The standard gradient tracking (GT) method in xin2018linear (blue curves, left axis) exhibits severe instability during the initial iterations, where both the loss and gradient norm rapidly explode. In contrast, Algorithm \ref{['alg:1']} (red curves, right axis) ensures a smooth decrease in both loss value and gradient norm. The zoom-in subplot highlights that both algorithms start from the same initialization.
• Figure 3: Comparison of loss and gradient norm evolution for Algorithm \ref{['alg:1']} and DGD with gradient clipping sunclipping on the a9a dataset.
• Figure 4: Comparison of the proposed algorithm with state-of-the-art methods in terms of training accuracy on the CIFAR-10 dataset.
• Figure 5: Comparison of the proposed algorithm with state-of-the-art methods in terms of test accuracy on the CIFAR-10 dataset.

Theorems & Definitions (28)

• Lemma 1: zhang2020improved, Lemma A.3
• Lemma 2: zhang2020improved, Corollary A.4
• Lemma 3
• Lemma 4: pu2020push, Lemma 1
• Lemma 5: pu2020push, Lemma 3
• Lemma 6: pu2020push, Lemma 4
• Lemma 7: pu2020push, Lemma 6
• Lemma 8
• proof
• Lemma 9