Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerated Methods with Complexity Separation Under Data Similarity for Federated Learning Problems

Dmitry Bylinkin, Sergey Skorik, Dmitriy Bystrov, Leonid Berezin, Aram Avetisyan, Aleksandr Beznosikov

TL;DR

The paper tackles federated optimization with a composite objective split into common and rare data components, and introduces Hessian similarity constants to enable complexity separation and reduce communication. It proposes SC-AccExtragradient, VRCS, and AccVRCS to achieve separated convergence rates for the two data-modes, with accelerated variants under convexity assumptions on $g$. The authors provide a rigorous sequence of proofs (stochastic, variance-reduced, and accelerated) and validate the methods on neural architectures (MLP on MNIST and ResNet-18 on CIFAR-10), demonstrating practical gains in communication efficiency in heterogeneous settings. Collectively, the work advances theory and practice for communication-efficient federated learning by explicitly decoupling the complexities arising from shared versus rare data.

Abstract

Heterogeneity within data distribution poses a challenge in many modern federated learning tasks. We formalize it as an optimization problem involving a computationally heavy composite under data similarity. By employing different sets of assumptions, we present several approaches to develop communication-efficient methods. An optimal algorithm is proposed for the convex case. The constructed theory is validated through a series of experiments across various problems.

Accelerated Methods with Complexity Separation Under Data Similarity for Federated Learning Problems

TL;DR

The paper tackles federated optimization with a composite objective split into common and rare data components, and introduces Hessian similarity constants to enable complexity separation and reduce communication. It proposes SC-AccExtragradient, VRCS, and AccVRCS to achieve separated convergence rates for the two data-modes, with accelerated variants under convexity assumptions on . The authors provide a rigorous sequence of proofs (stochastic, variance-reduced, and accelerated) and validate the methods on neural architectures (MLP on MNIST and ResNet-18 on CIFAR-10), demonstrating practical gains in communication efficiency in heterogeneous settings. Collectively, the work advances theory and practice for communication-efficient federated learning by explicitly decoupling the complexities arising from shared versus rare data.

Abstract

Heterogeneity within data distribution poses a challenge in many modern federated learning tasks. We formalize it as an optimization problem involving a computationally heavy composite under data similarity. By employing different sets of assumptions, we present several approaches to develop communication-efficient methods. An optimal algorithm is proposed for the convex case. The constructed theory is validated through a series of experiments across various problems.
Paper Structure (23 sections, 12 theorems, 153 equations, 2 figures, 6 algorithms)

This paper contains 23 sections, 12 theorems, 153 equations, 2 figures, 6 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Composite Optimization
  4. Similarity
  5. Our Contribution
  6. Setting
  7. Notation
  8. Assumptions
  9. Complexity Separation via SGD
  10. Discussion
  11. Complexity Separation via Variance Reduction
  12. Discussion
  13. Complexity Separation via Accelerated Extragradient
  14. Discussion
  15. Numerical Experiments
  16. ...and 8 more sections

Key Result

theorem 1

Consider Algorithm alg:stoch_extragrad for the problem prob_form_comp under Assumptions ass:h-ass:variances. Let the subproblem in Line eq:stoch_extragrad_subtask be solved approximately: Then the complexities in terms of communication rounds are for the nodes from $M_f$, $M_g$ respectively.

Figures (2)

  • Figure 1: Comparison of state-of-the-art distributed methods on equation \ref{['def:composite_crossentropy']} with $|M_f|=|M_g|=32$ and MNIST dataset. The criterion is the number of communication rounds over $M_f$. To show robustness, we vary the disparity parameter $\kappa$.
  • Figure 2: Comparison of Accelerated Extragradient and SC-AccExtragradient on equation \ref{['def:composite_crossentropy']} with $|M_f|=|M_g|=5$ and CIFAR-10 dataset. The criterion is the number of communication rounds over $M_f$. To show robustness, we vary the disparity parameter $\kappa$.

Theorems & Definitions (19)

  • definition 1
  • theorem 1
  • theorem 2
  • theorem 3
  • theorem 4
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • theorem 5
  • proof
  • ...and 9 more