Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Cubic Regularized Newton Learning with Sparsification-amplified Differential Privacy

Wei Huo, Changxin Liu, Kemi Ding, Karl Henrik Johansson, Ling Shi

TL;DR

This work tackles privacy preservation and communication efficiency in federated learning by introducing DP FCRN, a second order cubic regularized Newton method augmented with Gaussian noise and random k sparsification. The approach exploits the reduced sensitivity from sparsification to lower the required noise for client level DP while achieving faster convergence than first order methods. A non asymptotic convergence bound shows the utility loss scales as O(k log(1/δ0) (L0+L1D)^2 / (ε^2 m^2 μ)) times a term depending on the initial suboptimality, with a two phase convergence behavior. Empirical results on a large logistic regression task demonstrate substantial improvements over Fed-SGD with DP, highlighting favorable privacy-utility-communication tradeoffs enabled by sparsification and second order updates.

Abstract

This paper investigates the use of the cubic-regularized Newton method within a federated learning framework while addressing two major concerns that commonly arise in federated learning: privacy leakage and communication bottleneck. We introduce a federated learning algorithm called Differentially Private Federated Cubic Regularized Newton (DP-FCRN). By leveraging second-order techniques, our algorithm achieves lower iteration complexity compared to first-order methods. We also incorporate noise perturbation during local computations to ensure privacy. Furthermore, we employ sparsification in uplink transmission, which not only reduces the communication costs but also amplifies the privacy guarantee. Specifically, this approach reduces the necessary noise intensity without compromising privacy protection. We analyze the convergence properties of our algorithm and establish the privacy guarantee. Finally, we validate the effectiveness of the proposed algorithm through experiments on a benchmark dataset.

Federated Cubic Regularized Newton Learning with Sparsification-amplified Differential Privacy

TL;DR

This work tackles privacy preservation and communication efficiency in federated learning by introducing DP FCRN, a second order cubic regularized Newton method augmented with Gaussian noise and random k sparsification. The approach exploits the reduced sensitivity from sparsification to lower the required noise for client level DP while achieving faster convergence than first order methods. A non asymptotic convergence bound shows the utility loss scales as O(k log(1/δ0) (L0+L1D)^2 / (ε^2 m^2 μ)) times a term depending on the initial suboptimality, with a two phase convergence behavior. Empirical results on a large logistic regression task demonstrate substantial improvements over Fed-SGD with DP, highlighting favorable privacy-utility-communication tradeoffs enabled by sparsification and second order updates.

Abstract

This paper investigates the use of the cubic-regularized Newton method within a federated learning framework while addressing two major concerns that commonly arise in federated learning: privacy leakage and communication bottleneck. We introduce a federated learning algorithm called Differentially Private Federated Cubic Regularized Newton (DP-FCRN). By leveraging second-order techniques, our algorithm achieves lower iteration complexity compared to first-order methods. We also incorporate noise perturbation during local computations to ensure privacy. Furthermore, we employ sparsification in uplink transmission, which not only reduces the communication costs but also amplifies the privacy guarantee. Specifically, this approach reduces the necessary noise intensity without compromising privacy protection. We analyze the convergence properties of our algorithm and establish the privacy guarantee. Finally, we validate the effectiveness of the proposed algorithm through experiments on a benchmark dataset.
Paper Structure (18 sections, 10 theorems, 71 equations, 2 figures, 1 table, 2 algorithms)

This paper contains 18 sections, 10 theorems, 71 equations, 2 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries and Problem Formulation
  3. Basic Setup
  4. Newton Methods with Cubic Regularization
  5. Threat Model and DP
  6. Problem Statement
  7. Main Algorithm
  8. Privacy Analysis
  9. Convergence Analysis
  10. Numerical Evaluation
  11. Experimental Setup
  12. Performance and Comparison with Fed-SGD
  13. Trade-off between Privacy and Utility
  14. Conclusion and Future Work
  15. Supporting Lemmas
  16. ...and 3 more sections

Key Result

Lemma 1

(Gaussian Mechanism (balle2018improving)) A Gaussian mechanism $\mathcal{G}$ for a vector-valued computation $r: \zeta \to \mathbb{R}^{d}$ is obtained by computing the function $r$ on the input data $\zeta_{i} \in \zeta$ and then adding random Gaussian noise perturbation $\nu \sim \mathcal{N}(0, \si The Gaussian mechanism $\mathcal{G}$ is $\left(\frac{\sqrt{2 \log (1.25/ \delta) }\Delta}{\sigma},

Figures (2)

  • Figure 1: Performance comparison between Fed-SGD with DP and DP-FCRN with $\varepsilon = 0.8$.
  • Figure 2: Performance comparison between Fed-SGD and DP-FCRN under different DP parameters.

Theorems & Definitions (20)

  • Definition 1
  • Definition 2
  • Lemma 1
  • Definition 3
  • Remark 1
  • Remark 2
  • Theorem 1
  • Remark 3
  • Remark 4
  • Lemma 2
  • ...and 10 more