Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Locally Differentially Private Online Federated Learning With Correlated Noise

Jiaojiao Zhang, Linglingzhi Zhu, Dominik Fay, Mikael Johansson

TL;DR

This work addresses privacy-preserving online federated learning under streaming, non-IID data by introducing locally differentially private learning with temporally correlated noise. It extends matrix factorization (MF) based noise mechanisms to OFL with multiple local updates, and develops a perturbed-iterate analysis to bound the privacy-induced utility loss, yielding a dynamic regret bound for nonconvex losses under $(oldsymbol{ rac{ ext{epsilon}}{ ext{delta}}})$-LDP. The authors compare three MF realizations (binary tree, optimized factorization, Toeplitz) and show theoretically and empirically that correlated noise improves privacy-utility trade-offs versus independent noise, with static-regret results under strong convexity complementing the dynamic nonconvex analysis. Numerical experiments on logistic regression and CNNs (MNIST) corroborate the gains, highlighting reduced communication rounds via local updates and robust performance under privacy constraints. These results advance practical, privacy-aware OFL for dynamic, real-time applications like healthcare and recommendations.

Abstract

We introduce a locally differentially private (LDP) algorithm for online federated learning that employs temporally correlated noise to improve utility while preserving privacy. To address challenges posed by the correlated noise and local updates with streaming non-IID data, we develop a perturbed iterate analysis that controls the impact of the noise on the utility. Moreover, we demonstrate how the drift errors from local updates can be effectively managed for several classes of nonconvex loss functions. Subject to an $(ε,δ)$-LDP budget, we establish a dynamic regret bound that quantifies the impact of key parameters and the intensity of changes in the dynamic environment on the learning performance. Numerical experiments confirm the efficacy of the proposed algorithm.

Locally Differentially Private Online Federated Learning With Correlated Noise

TL;DR

This work addresses privacy-preserving online federated learning under streaming, non-IID data by introducing locally differentially private learning with temporally correlated noise. It extends matrix factorization (MF) based noise mechanisms to OFL with multiple local updates, and develops a perturbed-iterate analysis to bound the privacy-induced utility loss, yielding a dynamic regret bound for nonconvex losses under -LDP. The authors compare three MF realizations (binary tree, optimized factorization, Toeplitz) and show theoretically and empirically that correlated noise improves privacy-utility trade-offs versus independent noise, with static-regret results under strong convexity complementing the dynamic nonconvex analysis. Numerical experiments on logistic regression and CNNs (MNIST) corroborate the gains, highlighting reduced communication rounds via local updates and robust performance under privacy constraints. These results advance practical, privacy-aware OFL for dynamic, real-time applications like healthcare and recommendations.

Abstract

We introduce a locally differentially private (LDP) algorithm for online federated learning that employs temporally correlated noise to improve utility while preserving privacy. To address challenges posed by the correlated noise and local updates with streaming non-IID data, we develop a perturbed iterate analysis that controls the impact of the noise on the utility. Moreover, we demonstrate how the drift errors from local updates can be effectively managed for several classes of nonconvex loss functions. Subject to an -LDP budget, we establish a dynamic regret bound that quantifies the impact of key parameters and the intensity of changes in the dynamic environment on the learning performance. Numerical experiments confirm the efficacy of the proposed algorithm.

Paper Structure

This paper contains 27 sections, 7 theorems, 104 equations, 4 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Correlated noise
  5. Online distributed learning
  6. Dynamic regret for nonconvex problems
  7. Problem Formulation
  8. Algorithm
  9. Proposed Algorithm
  10. Adding Correlated Noise via MF
  11. Analysis
  12. Preliminaries
  13. Privacy-Utility Analysis
  14. Comparison of Correlated Noise and Independent Noise
  15. Static Regret Under Strongly Convex (SC) Condition
  16. ...and 12 more sections

Key Result

Lemma 4.6

The aggregated loss function $f^r$ satisfies the following implications: with $c_{\rm EB}=\mu_{{\rm QSC}}$ and $c_{\rm QG}=c_{\rm PL}/2$. If $f^r$ is $L$-smooth, then $\text{EB} \Rightarrow \text{P{\L}}$ with $c_{\rm PL}=c^2_{\rm EB}/L$.

Figures (4)

  • Figure 1: OFL framework
  • Figure 2: Binary tree mechanism
  • Figure 3: Comparison on logistic regression
  • Figure 4: Ablation and comparison on CNN classification under $(2,10^{-3})$-LDP budget

Theorems & Definitions (18)

  • Definition 2.1
  • Remark 3.1: Comparison with independent noise
  • Definition 4.5
  • Lemma 4.6: Theorem 2 in karimi2016linear
  • Corollary 4.7
  • proof
  • Lemma 4.8
  • proof
  • Lemma 4.9
  • proof
  • ...and 8 more