Setwise Coordinate Descent for Dual Asynchronous Decentralized Optimization

TL;DR

The paper tackles asynchronous decentralized optimization over networks by casting the dual problem and applying setwise coordinate descent with optimized neighbor selection. It develops SU-CD and SGS-CD as core schemes, extends to Lipschitz-aware variants SL-CD and SGSL-CD, and introduces online estimation of coordinate Lipschitz constants (SeL-CD and SGSeL-CD). Theoretical guarantees provide linear convergence; SGS-CD can yield up to a factor of $N_{\max}$ speedup over SU-CD, with Lipschitz-informed variants further accelerating convergence. Empirical results on decentralized and parallel settings validate the speedups and illustrate trade-offs with communication overhead and asynchronous operation, offering a practical framework for fast, asynchronous distributed optimization.

Abstract

In decentralized optimization over networks, synchronizing the updates of all nodes incurs significant communication overhead. For this reason, much of the recent literature has focused on the analysis and design of asynchronous optimization algorithms where nodes can activate anytime and contact a single neighbor to complete an iteration together. However, most works assume that the neighbor selection is done at random based on a fixed probability distribution (e.g., uniform), a choice that ignores the optimization landscape at the moment of activation. Instead, in this work we introduce an optimization-aware selection rule that chooses the neighbor providing the highest dual cost improvement (a quantity related to a dualization of the problem based on consensus). This scheme is related to the coordinate descent (CD) method with the Gauss-Southwell (GS) rule for coordinate updates; in our setting however, only a subset of coordinates is accessible at each iteration (because each node can communicate only with its neighbors), so the existing literature on GS methods does not apply. To overcome this difficulty, we develop a new analytical framework for smooth and strongly convex functions that covers our new class of setwise CD algorithms -- a class that applies to both decentralized and parallel distributed computing scenarios -- and we show that the proposed setwise GS rule can speed up the convergence in terms of iterations by a factor equal to the size of the largest coordinate set. We analyze extensions of these algorithms that exploit the knowledge of the smoothness constants when available and otherwise propose an algorithm to estimate these constants. Finally, we validate our theoretical results through extensive simulations.

Figures (8)

• Figure 1: Example of sets $\mathcal{S}_i$ and one possibility for $\mathcal{S}_i'$ and $\overline{\mathcal{S}_i'}$
• Figure 2: Comparison of the convergence rates of SU-CD and SGS-CD for quadratic problems in two settings: decentralized optimization over a network (left plots), and parallel distributed computation with parameter server (right plots).
• Figure 3: Comparison of SU-CD and SGS-CD for increasing $n$.
• Figure 4: Comparison of the performance of SL-CD (using global exact $L_i$ values) and SeL-CD in the parallel distributed setting. The experiment was designed to show that which of the two methods is faster depends on the problem considered.
• Figure 5: Performance of the algorithms presented in an LLS problem and two random graphs with different numbers of edges (left and right columns). Top plots: convergence in terms of the number of iterations. Bottom plots: convergence in terms of the number of vectors in $\mathbb{R}^d$ transmitted.

Theorems & Definitions (31)

• Lemma 1
• proof
• Lemma 2: Appendix C of hendrikx2019accelerated
• Definition 1: Setwise CD algorithm
• Remark 1
• Remark 2
• Lemma 3
• proof
• Theorem 1: Rate of SU-CD
• proof