Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedNS: A Fast Sketching Newton-Type Algorithm for Federated Learning

Jian Li, Yong Liu, Wei Wang, Haoran Wu, Weiping Wang

TL;DR

This paper approximates the centralized Newton's method by communicating the sketched square-root Hessian instead of the exact Hessian instead of the exact Hessian, and reduces the sketch size to match the effective dimension of the Hessian matrix.

Abstract

Recent Newton-type federated learning algorithms have demonstrated linear convergence with respect to the communication rounds. However, communicating Hessian matrices is often unfeasible due to their quadratic communication complexity. In this paper, we introduce a novel approach to tackle this issue while still achieving fast convergence rates. Our proposed method, named as Federated Newton Sketch methods (FedNS), approximates the centralized Newton's method by communicating the sketched square-root Hessian instead of the exact Hessian. To enhance communication efficiency, we reduce the sketch size to match the effective dimension of the Hessian matrix. We provide convergence analysis based on statistical learning for the federated Newton sketch approaches. Specifically, our approaches reach super-linear convergence rates w.r.t. the communication rounds for the first time. We validate the effectiveness of our algorithms through various experiments, which coincide with our theoretical findings.

FedNS: A Fast Sketching Newton-Type Algorithm for Federated Learning

TL;DR

This paper approximates the centralized Newton's method by communicating the sketched square-root Hessian instead of the exact Hessian instead of the exact Hessian, and reduces the sketch size to match the effective dimension of the Hessian matrix.

Abstract

Recent Newton-type federated learning algorithms have demonstrated linear convergence with respect to the communication rounds. However, communicating Hessian matrices is often unfeasible due to their quadratic communication complexity. In this paper, we introduce a novel approach to tackle this issue while still achieving fast convergence rates. Our proposed method, named as Federated Newton Sketch methods (FedNS), approximates the centralized Newton's method by communicating the sketched square-root Hessian instead of the exact Hessian. To enhance communication efficiency, we reduce the sketch size to match the effective dimension of the Hessian matrix. We provide convergence analysis based on statistical learning for the federated Newton sketch approaches. Specifically, our approaches reach super-linear convergence rates w.r.t. the communication rounds for the first time. We validate the effectiveness of our algorithms through various experiments, which coincide with our theoretical findings.
Paper Structure (20 sections, 3 theorems, 30 equations, 2 figures, 2 tables, 2 algorithms)

This paper contains 20 sections, 3 theorems, 30 equations, 2 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Backgrounds
  4. Centralized Newton's Method
  5. Newton's Method with Partial Sketching
  6. Federated Learning with Newton's Methods
  7. Federated Newton's Method (FedNewton)
  8. Federated Learning with Newton Sketch (FedNS)
  9. Federated Newton's Method with Dimension-Efficient Sketching (FedNDES)
  10. Theoretical Guarantees
  11. Convergence Analysis for FedNS
  12. Convergence Analysis for FedNDES
  13. Compared with Related Work
  14. Experiments
  15. Conclusion
  16. ...and 5 more sections

Key Result

Theorem 1

Let $\delta \in (0, 1)$. Under Assumptions asm.differentiable, asm.convex, asm.lipschitz, FedNS updates in Algorithm alg.FedNS based on an appropriate initialization $\|{\boldsymbol w}_0 - {\boldsymbol w}_{\mathcal{D},\lambda}\|_2 \leq \frac{\delta \gamma}{8 G}$. Using the iteration-dependent sketch This guarantee a super-linear convergence rate, since $\lim_{t \to \infty} \frac{\|{\boldsymbol w}_

Figures (2)

  • Figure 1: The loss discrepancy between the compared methods and the optimal learner in terms of the number of communication rounds $t$.
  • Figure 2: The loss discrepancy between the compared methods and the optimal learner in terms of the sketch size $k$ on the datasets cod-rna, covtype, SUSY, and phishing

Theorems & Definitions (7)

  • Theorem 1: Convergence guarantees of FedNS
  • Definition 1: Empirical effective dimension
  • Theorem 2: Convergence guarantees of FedNDES
  • proof : Proof of Theorem \ref{['thm.FedNS']}
  • proof : Proof of Theorem \ref{['thm.FedNDES']}
  • Theorem 3: Excess risk bound for FedNS with the squared loss
  • proof : Proof of Theorem \ref{['thm.KRR']}