Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lightweight Federated Learning with Differential Privacy and Straggler Resilience

Shu Hong, Xiaojun Lin, Lingjie Duan

TL;DR

This work tackles privacy in federated learning by addressing the dual challenges of differential privacy guarantees and straggler resilience without incurring prohibitive communication or computation costs. It introduces LightDP-FL, a lightweight scheme that injects both individual Gaussian noise and pairwise Gaussian noise, generated via Diffie-Hellman-based seeds, to mask local updates; the server largely cancels pairwise masks, while stragglers and colluders are accounted for in the noise design to maintain DP in the worst case. Theoretical analysis derives sufficient conditions for $(\varepsilon,\delta)$-DP under a joint Gaussian disturbance model and provides convergence bounds along with an optimization framework to select noise variances that balance privacy and accuracy. Experimental results on CIFAR-10 with ResNet-18 show that LightDP-FL achieves faster convergence and stronger straggler resilience than baselines at the same privacy level, while incurring much lower overhead than SMPC-based approaches. These findings demonstrate the practicality of provable DP in FL with minimal communication/computation burden and robust performance in imperfect network conditions. The approach offers a scalable path to deploy privacy-preserving FL in real-world settings where stragglers and untrusted servers are common and strict privacy guarantees are required.

Abstract

Federated learning (FL) enables collaborative model training through model parameter exchanges instead of raw data. To avoid potential inference attacks from exchanged parameters, differential privacy (DP) offers rigorous guarantee against various attacks. However, conventional methods of ensuring DP by adding local noise alone often result in low training accuracy. Combining secure multi-party computation (SMPC) with DP, while improving the accuracy, incurs high communication and computation overheads as well as straggler vulnerability, in either client-to-server or client-to-client links. In this paper, we propose LightDP-FL, a novel lightweight scheme that ensures provable DP against untrusted peers and server, while maintaining straggler resilience, low overheads and high training accuracy. Our scheme incorporates both individual and pairwise noise into each client's parameter, which can be implemented with minimal overheads. Given the uncertain straggler and colluder sets, we utilize the upper bound on the numbers of stragglers and colluders to prove sufficient noise variance conditions to ensure DP in the worst case. Moreover, we optimize the expected convergence bound to ensure accuracy performance by flexibly controlling the noise variances. Using the CIFAR-10 dataset, our experimental results demonstrate that LightDP-FL achieves faster convergence and stronger straggler resilience compared to baseline methods of the same DP level.

Lightweight Federated Learning with Differential Privacy and Straggler Resilience

TL;DR

This work tackles privacy in federated learning by addressing the dual challenges of differential privacy guarantees and straggler resilience without incurring prohibitive communication or computation costs. It introduces LightDP-FL, a lightweight scheme that injects both individual Gaussian noise and pairwise Gaussian noise, generated via Diffie-Hellman-based seeds, to mask local updates; the server largely cancels pairwise masks, while stragglers and colluders are accounted for in the noise design to maintain DP in the worst case. Theoretical analysis derives sufficient conditions for -DP under a joint Gaussian disturbance model and provides convergence bounds along with an optimization framework to select noise variances that balance privacy and accuracy. Experimental results on CIFAR-10 with ResNet-18 show that LightDP-FL achieves faster convergence and stronger straggler resilience than baselines at the same privacy level, while incurring much lower overhead than SMPC-based approaches. These findings demonstrate the practicality of provable DP in FL with minimal communication/computation burden and robust performance in imperfect network conditions. The approach offers a scalable path to deploy privacy-preserving FL in real-world settings where stragglers and untrusted servers are common and strict privacy guarantees are required.

Abstract

Federated learning (FL) enables collaborative model training through model parameter exchanges instead of raw data. To avoid potential inference attacks from exchanged parameters, differential privacy (DP) offers rigorous guarantee against various attacks. However, conventional methods of ensuring DP by adding local noise alone often result in low training accuracy. Combining secure multi-party computation (SMPC) with DP, while improving the accuracy, incurs high communication and computation overheads as well as straggler vulnerability, in either client-to-server or client-to-client links. In this paper, we propose LightDP-FL, a novel lightweight scheme that ensures provable DP against untrusted peers and server, while maintaining straggler resilience, low overheads and high training accuracy. Our scheme incorporates both individual and pairwise noise into each client's parameter, which can be implemented with minimal overheads. Given the uncertain straggler and colluder sets, we utilize the upper bound on the numbers of stragglers and colluders to prove sufficient noise variance conditions to ensure DP in the worst case. Moreover, we optimize the expected convergence bound to ensure accuracy performance by flexibly controlling the noise variances. Using the CIFAR-10 dataset, our experimental results demonstrate that LightDP-FL achieves faster convergence and stronger straggler resilience compared to baseline methods of the same DP level.

Paper Structure

This paper contains 29 sections, 5 theorems, 30 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Motivations
  3. Contributions
  4. Related Work
  5. Secure Multi-Party Computation (SMPC) Schemes
  6. Differential Privacy (DP)
  7. System Model and Problem Statement
  8. System Model
  9. Attack Model
  10. Problem Statement
  11. The Proposed LightDP-FL Scheme
  12. Generation of Pairwise Noise
  13. Local Parameter Updates
  14. Global Aggregation
  15. Theoretical Analysis
  16. ...and 14 more sections

Key Result

Lemma 1

Let be an $|\mathcal{I}_1| \times |\mathcal{I}_1|$ covariance matrix with for $i \in \mathcal{I}_1$ and for $i,j \in \mathcal{I}_1$, $i \neq j$, then the multivariate Gaussian distribution for $(\bm{m}_1, \cdots, \bm{m}_{|\mathcal{I}_1|})$ is where $det(\cdot)$ is the determinant of a matrix, $C_{\bm{m}}^{-1}$ is the inverse of $C_{\bm{m}}$, $\mathbf{X}=[\bm{x}_1, \cdots, \bm{x}_{|\mathcal{I}_

Figures (4)

  • Figure 1: Illustration of summed noise terms $\bm{n}_D$ in (\ref{['Equ:n_D']}) and $\bm{m}_D$ in (\ref{['Equ:m_D']}). 1) Due to the straggler issue, pairwise noise terms $\bm{r}_{ij}, \forall i \in \mathcal{N} \setminus \mathcal{N}_{\text{S}},j \in \mathcal{N}_{\text{S}}$ remain uncanceled in the aggregated noise term $\bm{n}_D$ in (\ref{['Equ:n_D']}), represented by red lines. 2) Remaining colluders $j \in \mathcal{N}_{\text{C}}\setminus(\mathcal{N}_{\text{C}} \cap \mathcal{N}_{\text{S}})$ will disclose their individual terms $\bm{n}_j$ and pairwise terms $\bm{r}_{ij}$'s. Hence, only $\bm{r}_{ij}$'s with $i \in \mathcal{I}_1=\{\mathcal{N}\setminus (\mathcal{N}_{\text{C}} \cup \mathcal{N}_{\text{S}}) \}$ and $j \in \mathcal{I}_2= \{\mathcal{N}_{\text{S}}\setminus (\mathcal{N}_{\text{C}} \cap \mathcal{N}_{\text{S}}) \}$, and noise terms $\bm{n}_j$'s with $j \in \mathcal{I}_1$ remain in the global disturbance term $\bm{m}_D$ for privacy.
  • Figure 2: Illustration of derived noise levels $(\sigma_\text{K},\sigma_{\text{U}})$ in Example \ref{['ex:variance sol-homo']}.
  • Figure 3: Test accuracy with the CIFAR-10 dataset using a ResNet-18 model under LightDP-FL, the SMPC+DP scheme, and the vanilla local noise-adding scheme for different privacy budgets $\varepsilon=3, 6,$ and $9$. We set $N=50$, $\bar{C}=10$, $\bar{S}=10$.
  • Figure 4: Performance comparison of LightDP-FL with the combination of HE and DP-noise adding for test accuracy of training a ResNet-18 model on the CIFAR-10 dataset.

Theorems & Definitions (7)

  • Definition 1: Differential Privacy
  • Lemma 1
  • Lemma 2
  • Proposition 1
  • Proposition 2
  • Proposition 3
  • Example 1