Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Analysis of regularized federated learning

Langming Liu, Dingxuan Zhou

TL;DR

This paper improves an algorithm called Loopless Local Gradient Descent which has advantages in reducing the expected communications by controlling a probability level by allowing flexible step sizes and carrying out novel analysis for the convergence of the algorithm in a non-convex setting in addition to the standard strongly convex setting.

Abstract

Federated learning is an efficient machine learning tool for dealing with heterogeneous big data and privacy protection. Federated learning methods with regularization can control the level of communications between the central and local machines. Stochastic gradient descent is often used for implementing such methods on heterogeneous big data, to reduce the communication costs. In this paper, we consider such an algorithm called Loopless Local Gradient Descent which has advantages in reducing the expected communications by controlling a probability level. We improve the method by allowing flexible step sizes and carry out novel analysis for the convergence of the algorithm in a non-convex setting in addition to the standard strongly convex setting. In the non-convex setting, we derive rates of convergence when the smooth objective function satisfies a Polyak-Łojasiewicz condition. When the objective function is strongly convex, a sufficient and necessary condition for the convergence in expectation is presented.

Analysis of regularized federated learning

TL;DR

This paper improves an algorithm called Loopless Local Gradient Descent which has advantages in reducing the expected communications by controlling a probability level by allowing flexible step sizes and carrying out novel analysis for the convergence of the algorithm in a non-convex setting in addition to the standard strongly convex setting.

Abstract

Federated learning is an efficient machine learning tool for dealing with heterogeneous big data and privacy protection. Federated learning methods with regularization can control the level of communications between the central and local machines. Stochastic gradient descent is often used for implementing such methods on heterogeneous big data, to reduce the communication costs. In this paper, we consider such an algorithm called Loopless Local Gradient Descent which has advantages in reducing the expected communications by controlling a probability level. We improve the method by allowing flexible step sizes and carry out novel analysis for the convergence of the algorithm in a non-convex setting in addition to the standard strongly convex setting. In the non-convex setting, we derive rates of convergence when the smooth objective function satisfies a Polyak-Łojasiewicz condition. When the objective function is strongly convex, a sufficient and necessary condition for the convergence in expectation is presented.

Paper Structure

This paper contains 21 sections, 11 theorems, 131 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem statement
  4. Algorithm: L2GDV
  5. Theoretical Results
  6. Convergence results in a non-convex situation
  7. Convergence results in a convex situation
  8. Experiments
  9. Experimental details
  10. Experimental results
  11. Non-convex situation
  12. Convex situation
  13. Conclusion
  14. Proof of Results in the Non-convex Situation
  15. General analysis for the non-convex situation
  16. ...and 6 more sections

Key Result

Theorem 1

Under Assumption assumption_0, when $0<\alpha\le\frac{\mu^2}{2\zeta L_F}$, the sequence $\{x^k\}$ defined by (L2GDnew) with $\alpha_k \equiv \alpha$ satisfies where $L_F :=\frac{L + \lambda}{n}, \sigma_{m}^2=\max\limits_{x \in\mathcal{X}^*}\{\sigma_{x}^2\}$, $\zeta = \frac{(1+2p)L^2}{(1-p)n^2}+\frac{(3-2p)\lambda^2}{pn^2}$, and

Figures (2)

  • Figure 1: Test set accuracy and training set loss w.r.t. communication rounds in a non-convex situation (i.e., using the CNN model). Figures (a) and (b) show the results for MNIST IID, while Figures (c) and (d) show those for MNIST Non-IID.
  • Figure 2: Test set accuracy and training set loss w.r.t. communication rounds in a convex situation (i.e., using the LR model). Figures (a) and (b) show the results for MNIST IID, while Figures (c) and (d) show those for MNIST Non-IID.

Theorems & Definitions (21)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Lemma 1
  • proof
  • Lemma 2
  • Lemma 3
  • proof
  • proof : Proof of Theorem \ref{['theorem_PL_fixed']}
  • ...and 11 more