Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerated Stochastic ExtraGradient: Mixing Hessian and Gradient Similarity to Reduce Communication in Distributed and Federated Learning

Dmitry Bylinkin, Kirill Degtyarev, Aleksandr Beznosikov

TL;DR

This work tackles the communication bottleneck in distributed and federated learning by exploiting Hessian similarity across clients and introducing a stochastic variant of Accelerated ExtraGradient (ASEG) that samples nodes and injects additive noise for privacy. The method delivers a communication-efficient scheme with provable convergence to a neighborhood under standard smoothness and convexity assumptions, and refined bounds when gradient similarity is incorporated. Theoretical results characterize how node sampling and noise affect convergence, and the subproblem is analyzed via SGD, SVRG, and loopless strategies, showing practical trade-offs. Empirical results on real datasets demonstrate favorable communication complexity and robustness to noise, highlighting ASEG as a promising approach for large-scale, privacy-preserving distributed optimization.

Abstract

Modern realities and trends in learning require more and more generalization ability of models, which leads to an increase in both models and training sample size. It is already difficult to solve such tasks in a single device mode. This is the reason why distributed and federated learning approaches are becoming more popular every day. Distributed computing involves communication between devices, which requires solving two key problems: efficiency and privacy. One of the most well-known approaches to combat communication costs is to exploit the similarity of local data. Both Hessian similarity and homogeneous gradients have been studied in the literature, but separately. In this paper, we combine both of these assumptions in analyzing a new method that incorporates the ideas of using data similarity and clients sampling. Moreover, to address privacy concerns, we apply the technique of additional noise and analyze its impact on the convergence of the proposed method. The theory is confirmed by training on real datasets.

Accelerated Stochastic ExtraGradient: Mixing Hessian and Gradient Similarity to Reduce Communication in Distributed and Federated Learning

TL;DR

This work tackles the communication bottleneck in distributed and federated learning by exploiting Hessian similarity across clients and introducing a stochastic variant of Accelerated ExtraGradient (ASEG) that samples nodes and injects additive noise for privacy. The method delivers a communication-efficient scheme with provable convergence to a neighborhood under standard smoothness and convexity assumptions, and refined bounds when gradient similarity is incorporated. Theoretical results characterize how node sampling and noise affect convergence, and the subproblem is analyzed via SGD, SVRG, and loopless strategies, showing practical trade-offs. Empirical results on real datasets demonstrate favorable communication complexity and robustness to noise, highlighting ASEG as a promising approach for large-scale, privacy-preserving distributed optimization.

Abstract

Modern realities and trends in learning require more and more generalization ability of models, which leads to an increase in both models and training sample size. It is already difficult to solve such tasks in a single device mode. This is the reason why distributed and federated learning approaches are becoming more popular every day. Distributed computing involves communication between devices, which requires solving two key problems: efficiency and privacy. One of the most well-known approaches to combat communication costs is to exploit the similarity of local data. Both Hessian similarity and homogeneous gradients have been studied in the literature, but separately. In this paper, we combine both of these assumptions in analyzing a new method that incorporates the ideas of using data similarity and clients sampling. Moreover, to address privacy concerns, we apply the technique of additional noise and analyze its impact on the convergence of the proposed method. The theory is confirmed by training on real datasets.
Paper Structure (18 sections, 16 theorems, 102 equations, 15 figures, 2 algorithms)

This paper contains 18 sections, 16 theorems, 102 equations, 15 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Distributed Methods under Similarity
  4. Privacy Preserving
  5. Stochastic Optimization
  6. Our Contribution
  7. Stochastic Version of Accelerated ExtraGradient
  8. Theoretical Analysis
  9. Analysis of the Subprolem
  10. SGD Approach
  11. SVRG Approach
  12. Loopless Approach
  13. Numerical Experiments
  14. Conclusion
  15. Lemma 1
  16. ...and 3 more sections

Key Result

Proposition 1

Consider $\delta$-relatedness $r_m(\cdot)$ to $r(\cdot)$ (Definition def:sim) for each $m\in[1,M]$. In this case, we have where $\sigma_{sim}^2=2\max_{m\in[1,M]}\{\|\nabla r_m(x_*)\|^2\}$ and $x_*$ is the solution of problem (prob_form).

Figures (15)

  • Figure 1: ASEG with different batch sizes
  • Figure 2: ASEG vs. AccSVRS on data with $\delta=10.15$ for quadratic task and $\delta=1.45$ for logistic one
  • Figure 3: ASEG vs. AccSVRS on data with $\delta=0.064$ for quadratic task and $\delta=0.061$ for logistic one
  • Figure 4: Stability of ASEG with SVRG-solver and uniform noise. The comparison is made solving quadratic problem on $M=200$ nodes
  • Figure 5: Stability of ASEG with SVRG-solver and uniform noise. The comparison is made solving logistic problem on $M=200$ nodes
  • ...and 10 more figures

Theorems & Definitions (29)

  • Definition 1
  • Definition 2
  • Definition 3
  • Proposition 1
  • proof
  • Remark 1
  • Lemma 1
  • Theorem 1
  • Corollary 1
  • Theorem 2
  • ...and 19 more