Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Private and Communication-Efficient Federated Learning based on Differentially Private Sketches

Meifan Zhang, Zhanhong Xie, Lihua Yin

TL;DR

An enhanced method is proposed, DPSFL-AC, which employs an adaptive clipping strategy, which employs an adaptive clipping strategy for differential privacy in federated learning.

Abstract

Federated learning (FL) faces two primary challenges: the risk of privacy leakage due to parameter sharing and communication inefficiencies. To address these challenges, we propose DPSFL, a federated learning method that utilizes differentially private sketches. DPSFL compresses the local gradients of each client using a count sketch, thereby improving communication efficiency, while adding noise to the sketches to ensure differential privacy (DP). We provide a theoretical analysis of privacy and convergence for the proposed method. Gradient clipping is essential in DP learning to limit sensitivity and constrain the addition of noise. However, clipping introduces bias into the gradients, negatively impacting FL performance. To mitigate the impact of clipping, we propose an enhanced method, DPSFL-AC, which employs an adaptive clipping strategy. Experimental comparisons with existing techniques demonstrate the superiority of our methods concerning privacy preservation, communication efficiency, and model accuracy.

Private and Communication-Efficient Federated Learning based on Differentially Private Sketches

TL;DR

An enhanced method is proposed, DPSFL-AC, which employs an adaptive clipping strategy, which employs an adaptive clipping strategy for differential privacy in federated learning.

Abstract

Federated learning (FL) faces two primary challenges: the risk of privacy leakage due to parameter sharing and communication inefficiencies. To address these challenges, we propose DPSFL, a federated learning method that utilizes differentially private sketches. DPSFL compresses the local gradients of each client using a count sketch, thereby improving communication efficiency, while adding noise to the sketches to ensure differential privacy (DP). We provide a theoretical analysis of privacy and convergence for the proposed method. Gradient clipping is essential in DP learning to limit sensitivity and constrain the addition of noise. However, clipping introduces bias into the gradients, negatively impacting FL performance. To mitigate the impact of clipping, we propose an enhanced method, DPSFL-AC, which employs an adaptive clipping strategy. Experimental comparisons with existing techniques demonstrate the superiority of our methods concerning privacy preservation, communication efficiency, and model accuracy.
Paper Structure (24 sections, 12 theorems, 7 equations, 7 figures, 1 table, 4 algorithms)

This paper contains 24 sections, 12 theorems, 7 equations, 7 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Differentially Private Federated Learning (DP+FL)
  4. Communication-efficient Federated Learning
  5. Preliminaries
  6. Differential Privacy
  7. FetchSGD rothchild2020fetchsgd
  8. FL based on DP Sketches (DPSFL)
  9. The framework of DPSFL
  10. Privacy Analysis
  11. Convergence Analysis of DPSFL
  12. Convergence of DPSFL-NonNoise
  13. Convergence of DPSFL
  14. DPSFL-AC: DPSFL with Adaptive Clipping
  15. Adaptive Clipping
  16. ...and 9 more sections

Key Result

Lemma 1

Composition of zCDP. Suppose $\mathcal{A}$ satisfies $\rho$- zCDP and $\mathcal{A}'$ satisfies $\rho'$-zCDP, and their composition $\mathcal{A}"(D)=\mathcal{A}'(D,\mathcal{A}(D))$ satisfies $(\rho+\rho')$-zCDP.

Figures (7)

  • Figure 1: Overview of the DPSFL-AC framework.
  • Figure 2: Gradient compression based on count sketch.
  • Figure 3: The impact of clipping
  • Figure 4: Comparison of Utility.
  • Figure 5: Comparison of communication efficiency.
  • ...and 2 more figures

Theorems & Definitions (17)

  • Definition 1
  • Definition 2
  • Definition 3
  • Lemma 1
  • Lemma 2
  • Theorem 1
  • Definition 4
  • Theorem 2
  • Lemma 3
  • Definition 5
  • ...and 7 more