Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Provable Acceleration of Distributed Optimization with Local Updates

Zuang Wang, Yongqiang Wang

TL;DR

This work addresses whether multiple local updates can accelerate distributed optimization when gradients are exact. It introduces a PEP-based framework to analyze the DIGing algorithm with $\tau$ local updates between communications and derives exact worst-case bounds. The key finding is that local updates can accelerate convergence, with the maximal gain at $\tau=2$ and no additional benefit for larger $\tau$; the optimal step size typically scales roughly as $\alpha^* \propto 1/\tau$ for larger $\tau$. These insights are validated through synthetic linear regression and CNN experiments, offering practical guidelines for choosing the number of local updates and step sizes in distributed settings with exact gradients.

Abstract

In conventional distributed optimization, each agent performs a single local update between two communication rounds with its neighbors to synchronize solutions. Inspired by the success of using multiple local updates in federated learning, incorporating local updates into distributed optimization has recently attracted increasing attention. However, unlike federated learning, where multiple local updates can accelerate learning by improving gradient estimation under mini-batch settings, it remains unclear whether similar benefits hold in distributed optimization when gradients are exact. Moreover, existing theoretical results typically require reducing the step size when multiple local updates are employed, which can entirely offset any potential benefit of these additional local updates and obscure their true impact on convergence. In this paper, we focus on the classic DIGing algorithm and leverage the tight performance bounds provided by Performance Estimation Problems (PEP) to show that incorporating local updates can indeed accelerate distributed optimization. To the best of our knowledge, this is the first rigorous demonstration of such acceleration for a broad class of objective functions. Our analysis further reveals that, under an appropriate step size, performing only two local updates is sufficient to achieve the maximal possible improvement, and that additional local updates provide no further gains. Because more updates increase computational cost, these findings offer practical guidance for efficient implementation. Extensive experiments on both synthetic and real-world datasets corroborate the theoretical findings.

Provable Acceleration of Distributed Optimization with Local Updates

TL;DR

This work addresses whether multiple local updates can accelerate distributed optimization when gradients are exact. It introduces a PEP-based framework to analyze the DIGing algorithm with local updates between communications and derives exact worst-case bounds. The key finding is that local updates can accelerate convergence, with the maximal gain at and no additional benefit for larger ; the optimal step size typically scales roughly as for larger . These insights are validated through synthetic linear regression and CNN experiments, offering practical guidelines for choosing the number of local updates and step sizes in distributed settings with exact gradients.

Abstract

In conventional distributed optimization, each agent performs a single local update between two communication rounds with its neighbors to synchronize solutions. Inspired by the success of using multiple local updates in federated learning, incorporating local updates into distributed optimization has recently attracted increasing attention. However, unlike federated learning, where multiple local updates can accelerate learning by improving gradient estimation under mini-batch settings, it remains unclear whether similar benefits hold in distributed optimization when gradients are exact. Moreover, existing theoretical results typically require reducing the step size when multiple local updates are employed, which can entirely offset any potential benefit of these additional local updates and obscure their true impact on convergence. In this paper, we focus on the classic DIGing algorithm and leverage the tight performance bounds provided by Performance Estimation Problems (PEP) to show that incorporating local updates can indeed accelerate distributed optimization. To the best of our knowledge, this is the first rigorous demonstration of such acceleration for a broad class of objective functions. Our analysis further reveals that, under an appropriate step size, performing only two local updates is sufficient to achieve the maximal possible improvement, and that additional local updates provide no further gains. Because more updates increase computational cost, these findings offer practical guidance for efficient implementation. Extensive experiments on both synthetic and real-world datasets corroborate the theoretical findings.
Paper Structure (12 sections, 1 theorem, 6 equations, 5 figures, 1 algorithm)

This paper contains 12 sections, 1 theorem, 6 equations, 5 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. A Motivating Example
  3. System Model and Algorithm Description
  4. Assumptions
  5. Incorporating Local Updates in DIGing
  6. Main Results
  7. PEP Formulation
  8. PEP Results
  9. Numerical Experiments
  10. Linear Regression
  11. Training of Convolutional Neural Networks
  12. Conclusions

Key Result

Lemma 1

(Interpolation for function class $\mathcal{F}_{\mu, L}$taylor2016) Let $I$ be any interpolation index set, and let $\{x^k, g^k, f^k\}_{k\in I} \subset \mathbb{R}^d\times\mathbb{R}^d\times\mathbb{R}$. There exists $f \in \mathcal{F}_{\mu, L}$ satisfying $f(x^k) = f^k$ and $g^k\in \partial f(x^k)$ fo

Figures (5)

  • Figure 1: Evolution of the optimization error under different numbers of local updates ($\tau$) and their respective optimal step sizes ($\alpha^\ast$). The optimal step sizes were obtained via a grid search with a resolution of 0.0001. The two objective functions are $f_1(x) = 0.5(x + 1)^2$ and $f_2(x) = 0.1(x - 1)^2$, with both agents initialized at zero. The mixing matrix is $0.50.50.50.5.$
  • Figure 2: PEP based exact worst-case convergence error of Algorithm \ref{['alg:diging_local']} under different numbers of local updates, each using its respective optimal step size $\alpha^\ast$, across various graph topologies. $\alpha^\ast$ for each $\tau$ was obtained via a grid search over $\alpha \in [0.01, 0.8]$ with 0.01 resolution for the function class $\mathcal{F}_{\mu, L}$ with $\mu = 0.1$ and $L = 1$. The number of agents is 4.
  • Figure 3: Algorithm \ref{['alg:diging_local']}'s exact worst-case convergence error at the final iteration $T$ as a function of the step size under $\tau = 4$ local updates and an all-to-all graph. The step size was grid-searched over $\alpha \in [0.01, 0.8]$ with a resolution of $0.01$.
  • Figure 4: DIGing-based training of the linear regression model under different numbers of local updates. The network consists of 4 agents for each topology. The step size for each $\tau$ was determined via a grid search over $\alpha \in [0.01, 0.8]$ with 0.01 resolution.
  • Figure 5: DIGing-based training of CNN on the MNIST dataset under different numbers of local updates. The network consists of 10 agents arranged in an all-to-all communication graph. The step size for each $\tau$ was determined via a grid search over $\alpha \in [0.01, 0.8]$ with 0.01 resolution. Each loss value is averaged over 10 runs, with error bars indicating the standard deviation.

Theorems & Definitions (1)

  • Lemma 1