Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

First Provable Guarantees for Practical Private FL: Beyond Restrictive Assumptions

Egor Shulgin, Grigory Malinovsky, Sarit Khirirat, Peter Richtárik

TL;DR

This paper addresses differential privacy in federated learning under realistic settings by eliminating restrictive assumptions such as bounded gradients or homogeneous data. It introduces Fed-$α$-NormEC, which combines smoothed normalization and EF21 error feedback with local updates and partial client participation, along with both GD and Incremental Gradient variants. The authors prove convergence for non-convex, smooth objectives and provide DP guarantees leveraging privacy amplification from subsampling; they also validate the approach on private deep learning tasks. The results demonstrate favorable privacy-utility trade-offs and reduced communication costs, making private FL more viable in practice.

Abstract

Federated Learning (FL) enables collaborative training on decentralized data. Differential privacy (DP) is crucial for FL, but current private methods often rely on unrealistic assumptions (e.g., bounded gradients or heterogeneity), hindering practical application. Existing works that relax these assumptions typically neglect practical FL features, including multiple local updates and partial client participation. We introduce Fed-$α$-NormEC, the first differentially private FL framework providing provable convergence and DP guarantees under standard assumptions while fully supporting these practical features. Fed-$α$-NormE integrates local updates (full and incremental gradient steps), separate server and client stepsizes, and, crucially, partial client participation, which is essential for real-world deployment and vital for privacy amplification. Our theoretical guarantees are corroborated by experiments on private deep learning tasks.

First Provable Guarantees for Practical Private FL: Beyond Restrictive Assumptions

TL;DR

This paper addresses differential privacy in federated learning under realistic settings by eliminating restrictive assumptions such as bounded gradients or homogeneous data. It introduces Fed--NormEC, which combines smoothed normalization and EF21 error feedback with local updates and partial client participation, along with both GD and Incremental Gradient variants. The authors prove convergence for non-convex, smooth objectives and provide DP guarantees leveraging privacy amplification from subsampling; they also validate the approach on private deep learning tasks. The results demonstrate favorable privacy-utility trade-offs and reduced communication costs, making private FL more viable in practice.

Abstract

Federated Learning (FL) enables collaborative training on decentralized data. Differential privacy (DP) is crucial for FL, but current private methods often rely on unrealistic assumptions (e.g., bounded gradients or heterogeneity), hindering practical application. Existing works that relax these assumptions typically neglect practical FL features, including multiple local updates and partial client participation. We introduce Fed--NormEC, the first differentially private FL framework providing provable convergence and DP guarantees under standard assumptions while fully supporting these practical features. Fed--NormE integrates local updates (full and incremental gradient steps), separate server and client stepsizes, and, crucially, partial client participation, which is essential for real-world deployment and vital for privacy amplification. Our theoretical guarantees are corroborated by experiments on private deep learning tasks.
Paper Structure (33 sections, 19 theorems, 130 equations, 4 figures, 1 algorithm)

This paper contains 33 sections, 19 theorems, 130 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Works
  4. Preliminaries
  5. Fed-$\alpha$-NormEC
  6. Convergence Results
  7. One Local GD Step
  8. Multiple Local GD Steps
  9. Extension to Local IG Steps
  10. Experiments
  11. Conclusion
  12. Additional experiments and details
  13. Additional details.
  14. Fed-$\alpha$-NormEC vs FedAvg
  15. Useful Lemmas
  16. ...and 18 more sections

Key Result

Theorem 1

Consider Fed-$\alpha$-NormEC for solving Problem eqn:problem where assum:smooth holds. Let $\beta,\alpha>0$ be chosen such that $\frac{\beta}{\alpha+R} < 1$ with $R = \max_{i \in [1,M]} \left\| v_i^0 - \frac{x^0 - \mathcal{T}_i(x^0)}{\gamma} \right\|$. If $\gamma = \frac{1}{2L}$ and $\eta \leq \mi where $B= 2\frac{(p-1)^2}{p} + 2(1-p) + 2\sigma^2_{\rm DP}/p$, and $\Delta^{\inf} = f^{\inf} - \fra

Figures (4)

  • Figure 1: Convergence of Fed-$\alpha$-NormEC under Full [solid] and Partial participation [dotted] for $p=0.25$.
  • Figure 2: Convergence of Fed-$\alpha$-NormEC across different partial participation rates. Horizontal axis takes into account the total number of transmissions from client to server.
  • Figure 3: Error Compensation (EC) provides significant benefits across various $\beta$ values.
  • Figure 4: The highest test accuracy achieved by FedAvg for different $\beta$ and $\gamma$ parameters.

Theorems & Definitions (37)

  • Definition 1: Client-level DP
  • Theorem 1: Fed-$\alpha$-NormEC with local GD steps
  • Corollary 1
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Lemma 4
  • ...and 27 more