DCatalyst: A Unified Accelerated Framework for Decentralized Optimization

TL;DR

This work introduces DCatalyst, a unified decentralized acceleration framework that injects Nesterov-style acceleration into a broad class of decentralized algorithms solving composite objectives $u(x)=f(x)+r(x)$. It develops inexact estimating sequences to manage consensus errors and subproblem inexactness, enabling rigorous convergence analysis for both strongly convex and convex cases. The framework is instantiated with multiple algorithms (e.g., SONATA, PUDA, PMGT-LSVRG), including accelerated variants that exploit function similarity and finite-sum structure to achieve near-optimal communication and computation complexities (up to log factors). Empirically, DCatalyst delivers substantial acceleration across diverse scenarios, especially for ill-conditioned problems, and provides practical tuning guidance for inner- and outer-loop iterations. Overall, the approach broadens the applicability of accelerated decentralized optimization, reduces communication bottlenecks, and enables VR and nonsmooth composite objectives within a unified theory and toolkit.

Abstract

We study decentralized optimization over a network of agents, modeled as graphs, with no central server. The goal is to minimize $f+r$, where $f$ represents a (strongly) convex function averaging the local agents' losses, and $r$ is a convex, extended-value function. We introduce DCatalyst, a unified black-box framework that integrates Nesterov acceleration into decentralized optimization algorithms. %, enhancing their performance. At its core, DCatalyst operates as an \textit{inexact}, \textit{momentum-accelerated} proximal method (forming the outer loop) that seamlessly incorporates any selected decentralized algorithm (as the inner loop). We demonstrate that DCatalyst achieves optimal communication and computational complexity (up to log-factors) across various decentralized algorithms and problem instances. Notably, it extends acceleration capabilities to problem classes previously lacking accelerated solution methods, thereby broadening the effectiveness of decentralized methods. On the technical side, our framework introduce the {\it inexact estimating sequences}--a novel extension of the well-known Nesterov's estimating sequences, tailored for the minimization of composite losses in decentralized settings. This method adeptly handles consensus errors and inexact solutions of agents' subproblems, challenges not addressed by existing models.

Figures (7)

• Figure 1: Comparison of distributed algorithms under strongly convex and non-smooth setting in (a): communication cost (left) and (b): computation cost (right).
• Figure 2: Comparison of distributed algorithms under strongly convex and non-smooth setting on the influence of parameters (a): similarity $\beta/\mu$ (left), (b): total sample size $N$ (middle) and (c): global condition number $\kappa_g$ to the number communication rounds needed to reach a precision of $10^{-4}$ (right).
• Figure 3: The comparison between APMli2020decentralized and DCatalyst-SONATA-L under convex and non-smooth setting in (a): communication cost (left) and (b): computation cost (right).
• Figure 4: Comparison between PMGT-LSVRGye2021pmgtvr and DCatalyst-PMGT-LSVRG, under the finite-sum setting, with strongly convex non-smooth $u$ and $n<<\kappa_s$. (a): communication cost (left); (b): computation cost (right).
• Figure 5: Comparison of distributed algorithms under finite-sum setting with a strongly convex smooth objective function in (a): communication cost (left) and (b): Computation cost (right).

Theorems & Definitions (35)

• Example 1
• Proposition 1: Convergence of the outer loop
• Proposition 2
• Theorem 3
• Proposition 4: Convergence of the outer loop
• Proposition 5
• Theorem 6
• Lemma 7
• Lemma 8
• proof