Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal Client Sampling in Federated Learning with Client-Level Heterogeneous Differential Privacy

Jiahao Xu, Rui Hu, Olivera Kotevska

TL;DR

This work tackles client-level differential privacy in federated learning with heterogeneous privacy budgets, addressing the inefficiency of enforcing a single global privacy level. It introduces GDPFed, which groups clients by privacy budgets and enforces DP within each group, yielding better utility under a robust honest-but-curious threat model; GDPFed$^+$ further improves performance through per-group sparsification and optimized client sampling. The authors provide per-group DP guarantees, a convergence bound that reveals the roles of model dimension and sampling in noise amplification, and a practical approach to selecting sparsification and sampling to minimize convergence error. Empirical results on multiple benchmarks show that GDPFed$^+$ surpasses state-of-the-art baselines in accuracy while maintaining strict group-level DP guarantees, highlighting its practicality for real-world cross-device FL with privacy heterogeneity.

Abstract

Federated Learning with client-level differential privacy (DP) provides a promising framework for collaboratively training models while rigorously protecting clients' privacy. However, classic approaches like DP-FedAvg struggle when clients have heterogeneous privacy requirements, as they must uniformly enforce the strictest privacy level across all clients, leading to excessive DP noise and significant degradation in model utility. Existing methods to improve the model utility in such heterogeneous privacy settings often assume a trusted server and are largely heuristic, resulting in suboptimal performance and lacking strong theoretical foundations. In this work, we address these challenges under a practical attack model where both clients and the server are honest-but-curious. We propose GDPFed, which partitions clients into groups based on their privacy budgets and achieves client-level DP within each group to reduce the privacy budget waste and hence improve the model utility. Based on the privacy and convergence analysis of GDPFed, we find that the magnitude of DP noise depends on both model dimensionality and the per-group client sampling ratios. To further improve the performance of GDPFed, we introduce GDPFed$^+$, which integrates model sparsification to eliminate unnecessary noise and optimizes per-group client sampling ratios to minimize convergence error. Extensive empirical evaluations on multiple benchmark datasets demonstrate the effectiveness of GDPFed$^+$, showing substantial performance gains compared with state-of-the-art methods.

Optimal Client Sampling in Federated Learning with Client-Level Heterogeneous Differential Privacy

TL;DR

This work tackles client-level differential privacy in federated learning with heterogeneous privacy budgets, addressing the inefficiency of enforcing a single global privacy level. It introduces GDPFed, which groups clients by privacy budgets and enforces DP within each group, yielding better utility under a robust honest-but-curious threat model; GDPFed further improves performance through per-group sparsification and optimized client sampling. The authors provide per-group DP guarantees, a convergence bound that reveals the roles of model dimension and sampling in noise amplification, and a practical approach to selecting sparsification and sampling to minimize convergence error. Empirical results on multiple benchmarks show that GDPFed surpasses state-of-the-art baselines in accuracy while maintaining strict group-level DP guarantees, highlighting its practicality for real-world cross-device FL with privacy heterogeneity.

Abstract

Federated Learning with client-level differential privacy (DP) provides a promising framework for collaboratively training models while rigorously protecting clients' privacy. However, classic approaches like DP-FedAvg struggle when clients have heterogeneous privacy requirements, as they must uniformly enforce the strictest privacy level across all clients, leading to excessive DP noise and significant degradation in model utility. Existing methods to improve the model utility in such heterogeneous privacy settings often assume a trusted server and are largely heuristic, resulting in suboptimal performance and lacking strong theoretical foundations. In this work, we address these challenges under a practical attack model where both clients and the server are honest-but-curious. We propose GDPFed, which partitions clients into groups based on their privacy budgets and achieves client-level DP within each group to reduce the privacy budget waste and hence improve the model utility. Based on the privacy and convergence analysis of GDPFed, we find that the magnitude of DP noise depends on both model dimensionality and the per-group client sampling ratios. To further improve the performance of GDPFed, we introduce GDPFed, which integrates model sparsification to eliminate unnecessary noise and optimizes per-group client sampling ratios to minimize convergence error. Extensive empirical evaluations on multiple benchmark datasets demonstrate the effectiveness of GDPFed, showing substantial performance gains compared with state-of-the-art methods.
Paper Structure (36 sections, 6 theorems, 3 equations, 2 figures, 8 tables, 2 algorithms)

This paper contains 36 sections, 6 theorems, 3 equations, 2 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. System Settings and Preliminary
  3. Attack Model
  4. Federated Learning and FedAvg
  5. Differential Privacy
  6. Related Work
  7. Client-level DP-FedAvg
  8. FL with Heterogeneous Privacy Preferences
  9. Federated Learning with Heterogeneous Group Client-Level DP
  10. GDPFed
  11. Privacy Analysis of GDPFed
  12. Analyzing DP Noise
  13. Sparsification-Amplified GDPFed with Optimal Client Sampling
  14. GDPFed with Per-group Sparsification
  15. Bounded Sparsification Error
  16. ...and 21 more sections

Key Result

Theorem 1

Suppose clients in group $m$ are sampled without replacement with probability $q_m$ at each round. For any $\epsilon_m < 2\log(1/\delta)$ and $\delta \in (0,1)$, GDPFed satisfies $(\epsilon_m, \delta)$-DP for clients in group $m$ after $T$ rounds if

Figures (2)

  • Figure 1: Optimal client sampling ratios under varying $\epsilon_1$. Sampling ratio for each group is adjusted dynamically to satisfy the global constraint with fixed $\epsilon_2$ and $\epsilon_3$.
  • Figure 2: Convergence curve comparison between P-FedAvg, DP-FedAvg, GDPFed, and GDPFed$^+$.

Theorems & Definitions (13)

  • Definition 1: $(\epsilon,\delta)$-DP dp_og
  • Theorem 1: Per-Group Privacy Guarantees of GDPFed
  • proof
  • Remark 1
  • Lemma 1: Parallel Composition of DP yu2019differentially
  • Corollary 1: Privacy Guarantee of GDPFed
  • Remark 2
  • Lemma 2: Post-Processing of DP dp_og
  • Lemma 3: Bounded Sparsification
  • Theorem 2: Convergence Result of GDPFed
  • ...and 3 more