Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentially Private Online Federated Learning with Correlated Noise

Jiaojiao Zhang, Linglingzhi Zhu, Mikael Johansson

TL;DR

This work addresses privacy-preserving online federated learning with streaming, non-iid data and adaptive continuous model releases. It introduces temporally correlated differential privacy noise and a perturbed-iterate analysis to bound utility while satisfying $(\epsilon,\delta)$-DP, leveraging matrix-factorization-based noise design to create correlations over time. The analysis yields dynamic regret bounds under QSC and static regret bounds under SC, explicitly characterizing the trade-offs among privacy, communication, and environmental change. Empirical results on logistic regression validate improved utility with reduced communication and demonstrate robustness to varying DP budgets. Overall, the approach advances private OFL by enabling adaptive releases with provable utility-privacy guarantees and practical performance gains.

Abstract

We introduce a novel differentially private algorithm for online federated learning that employs temporally correlated noise to enhance utility while ensuring privacy of continuously released models. To address challenges posed by DP noise and local updates with streaming non-iid data, we develop a perturbed iterate analysis to control the impact of the DP noise on the utility. Moreover, we demonstrate how the drift errors from local updates can be effectively managed under a quasi-strong convexity condition. Subject to an $(ε, δ)$-DP budget, we establish a dynamic regret bound over the entire time horizon, quantifying the impact of key parameters and the intensity of changes in dynamic environments. Numerical experiments confirm the efficacy of the proposed algorithm.

Differentially Private Online Federated Learning with Correlated Noise

TL;DR

This work addresses privacy-preserving online federated learning with streaming, non-iid data and adaptive continuous model releases. It introduces temporally correlated differential privacy noise and a perturbed-iterate analysis to bound utility while satisfying -DP, leveraging matrix-factorization-based noise design to create correlations over time. The analysis yields dynamic regret bounds under QSC and static regret bounds under SC, explicitly characterizing the trade-offs among privacy, communication, and environmental change. Empirical results on logistic regression validate improved utility with reduced communication and demonstrate robustness to varying DP budgets. Overall, the approach advances private OFL by enabling adaptive releases with provable utility-privacy guarantees and practical performance gains.

Abstract

We introduce a novel differentially private algorithm for online federated learning that employs temporally correlated noise to enhance utility while ensuring privacy of continuously released models. To address challenges posed by DP noise and local updates with streaming non-iid data, we develop a perturbed iterate analysis to control the impact of the DP noise on the utility. Moreover, we demonstrate how the drift errors from local updates can be effectively managed under a quasi-strong convexity condition. Subject to an -DP budget, we establish a dynamic regret bound over the entire time horizon, quantifying the impact of key parameters and the intensity of changes in dynamic environments. Numerical experiments confirm the efficacy of the proposed algorithm.
Paper Structure (18 sections, 4 theorems, 67 equations, 4 figures, 1 algorithm)

This paper contains 18 sections, 4 theorems, 67 equations, 4 figures, 1 algorithm.

Table of Contents

  1. INTRODUCTION
  2. PROBLEM FORMULATION
  3. ALGORITHM
  4. Proposed algorithm
  5. Matrix factorization
  6. ANALYSIS
  7. Utility
  8. Privacy
  9. Utility-privacy
  10. EXPERIMENTS
  11. Impact of $\tau$
  12. Comparison under different DP budgets
  13. CONCLUSIONS
  14. Fixed step size $\tilde{\eta}$
  15. QSC case: proof of Lemma \ref{['lem-u']}
  16. ...and 3 more sections

Key Result

Lemma IV.5

Under Assumptions asm-smooth-asm-regular, if $\tilde{\eta} \le \frac{1}{8L}$, for Algorithm alg-fl, we have where and $C_{R}:=\sum_{r=0}^{R-1}\mathbb{E}\|\operatorname{P}_{\mathcal{X}_r^{\star}}^{x^{r}}-\operatorname{P}_{\mathcal{X}^{\star}}^{x^{r}}\|^2$.

Figures (4)

  • Figure 1: Our OFL architecture
  • Figure 2: Test to show the validity of $\|{\bf B}\|_F^2=\mathcal{O}(R^2)$
  • Figure 3: Impact of $\tau$
  • Figure 4: Comparison under different DP budgets

Theorems & Definitions (6)

  • Definition II.1
  • Lemma IV.5
  • Theorem IV.6: Theorems 2.1 and 3.1 in denisov2022improved
  • Theorem IV.7
  • Corollary IV.8: Static regret under SC
  • Remark IV.9