Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DP$^2$-FedSAM: Enhancing Differentially Private Federated Learning Through Personalized Sharpness-Aware Minimization

Zhenxiao Zhang, Yuanxiong Guo, Yanmin Gong

TL;DR

A novel DPFL method named DP$^2$-FedSAM leverages personalized partial model-sharing and sharpness-aware minimization optimizer to mitigate the adverse impact of noise addition and clipping, thereby significantly improving model utility without sacrificing privacy.

Abstract

Federated learning (FL) is a distributed machine learning approach that allows multiple clients to collaboratively train a model without sharing their raw data. To prevent sensitive information from being inferred through the model updates shared in FL, differentially private federated learning (DPFL) has been proposed. DPFL ensures formal and rigorous privacy protection in FL by clipping and adding random noise to the shared model updates. However, the existing DPFL methods often result in severe model utility degradation, especially in settings with data heterogeneity. To enhance model utility, we propose a novel DPFL method named DP$^2$-FedSAM: Differentially Private and Personalized Federated Learning with Sharpness-Aware Minimization. DP$^2$-FedSAM leverages personalized partial model-sharing and sharpness-aware minimization optimizer to mitigate the adverse impact of noise addition and clipping, thereby significantly improving model utility without sacrificing privacy. From a theoretical perspective, we provide a rigorous theoretical analysis of the privacy and convergence guarantees of our proposed method. To evaluate the effectiveness of DP$^2$-FedSAM, we conduct extensive evaluations based on common benchmark datasets. Our results verify that our method improves the privacy-utility trade-off compared to the existing DPFL methods, particularly in heterogeneous data settings.

DP$^2$-FedSAM: Enhancing Differentially Private Federated Learning Through Personalized Sharpness-Aware Minimization

TL;DR

A novel DPFL method named DP-FedSAM leverages personalized partial model-sharing and sharpness-aware minimization optimizer to mitigate the adverse impact of noise addition and clipping, thereby significantly improving model utility without sacrificing privacy.

Abstract

Federated learning (FL) is a distributed machine learning approach that allows multiple clients to collaboratively train a model without sharing their raw data. To prevent sensitive information from being inferred through the model updates shared in FL, differentially private federated learning (DPFL) has been proposed. DPFL ensures formal and rigorous privacy protection in FL by clipping and adding random noise to the shared model updates. However, the existing DPFL methods often result in severe model utility degradation, especially in settings with data heterogeneity. To enhance model utility, we propose a novel DPFL method named DP-FedSAM: Differentially Private and Personalized Federated Learning with Sharpness-Aware Minimization. DP-FedSAM leverages personalized partial model-sharing and sharpness-aware minimization optimizer to mitigate the adverse impact of noise addition and clipping, thereby significantly improving model utility without sacrificing privacy. From a theoretical perspective, we provide a rigorous theoretical analysis of the privacy and convergence guarantees of our proposed method. To evaluate the effectiveness of DP-FedSAM, we conduct extensive evaluations based on common benchmark datasets. Our results verify that our method improves the privacy-utility trade-off compared to the existing DPFL methods, particularly in heterogeneous data settings.
Paper Structure (35 sections, 15 theorems, 63 equations, 7 figures, 3 tables, 2 algorithms)

This paper contains 35 sections, 15 theorems, 63 equations, 7 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Federated Learning
  4. SAM
  5. Differential Privacy
  6. Attack Model and Privacy Goal
  7. DP-FedAvg: Achieving Client-level DP in FL
  8. Methodology
  9. The Impact of Clipping
  10. Partial Model-Sharing Mitigates the Effect of Clipping
  11. SAM Mitigates the Effect of Clipping
  12. Mitigating the Effect of Adding Noise by Combining Partial Model-Sharing and SAM
  13. DP$^2$-FedSAM Algorithm
  14. Theoretical Analysis
  15. Convergence Analysis
  16. ...and 20 more sections

Key Result

Lemma 1

Let $f: \mathcal{D} \rightarrow \mathbb{R}^d$ be a query function with $\ell_2$-sensitivity $\psi(h)$. The Gaussian mechanism $\mathcal{M} = f(D) + \mathcal{N}(0, \sigma^2 \psi(f)^2 \bm{I}_d)$ satisfies $(\alpha,\alpha /2\sigma^2)$-RDP.

Figures (7)

  • Figure 1: Test accuracy on CIFAR-10 with a CNN for different methods under a non-IID data partition, where 1000 clients each has data from only 2 classes. P-FedSAM is essentially DP$^2$-FedSAM without the mechanisms of clipping and adding noise. DP$^2$-FedSAM exhibits enhanced robustness compared to DP-FedAvg.
  • Figure 2: Illustration of sharp and flat loss landscape. The flat minimum is more robust than the sharp one under the same perturbation in DP training.
  • Figure 3: An overview of DP$^2$-FedSAM. Partial model personalization allows each client to locally retain a personal classifier and only share the representation extractor with the server for aggregation. The server aggregates the shared representation extractor and sends it back to all clients. During local training, SAM is applied to enhance the robustness of the shared representation extractor.
  • Figure 4: Training performance versus communication round for FEMNIST and CIFAR-10 under $\epsilon= 1.0$.
  • Figure 5: The averaged norm of local updates $\Delta_i^t$ versus communication round.
  • ...and 2 more figures

Theorems & Definitions (24)

  • Definition 1: Client-level $(\epsilon,\delta)$-DP McMahan2018learning
  • Definition 2: $(\alpha,\rho)$-RDP mironov2017renyi
  • Definition 3: $\ell_2$-sensitivity dwork2014algorithmic
  • Lemma 1: Gaussian Mechanism mironov2017renyi
  • Lemma 2: From RDP to $(\epsilon, \delta)$-DP wang2019subsampled
  • Lemma 3: RDP Composition mironov2017renyi
  • Lemma 4: RDP for Subsampling Mechanism wang2019subsampled
  • Remark 1
  • Remark 2
  • Remark 3
  • ...and 14 more