Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Trade-offs: A Unified Framework for Privacy, Robustness, and Communication Efficiency in Federated Learning

Yue Xia, Tayyebeh Jahani-Nezhad, Rawad Bitar

TL;DR

The paper tackles the challenge of simultaneously ensuring privacy, robustness to Byzantine clients, and communication efficiency in federated learning. It introduces Fed-DPRoC, a framework that uses robust-compatible compression to preserve the guarantees of robust aggregation while reducing communication, and instantiates it as RobAJoL with JL-based compression and Gaussian DP. Theoretical results show that the JL transform preserves robustness with κ'=(1+ε_{JL})^2κ and δ_{RA}=ε_{JL}^2$, while achieving DP and reducing bidirectional communication from $O(d)$ to $O(k)$. Empirical evaluations on CIFAR-10, Fashion-MNIST, and FEMNIST demonstrate that RobAJoL outperforms state-of-the-art DP+robust schemes across attacks and privacy budgets, achieving strong robustness and utility with significantly lowered communication overhead.

Abstract

We propose Fed-DPRoC, a novel federated learning framework designed to jointly provide differential privacy (DP), Byzantine robustness, and communication efficiency. Central to our approach is the concept of robust-compatible compression, which allows reducing the bi-directional communication overhead without undermining the robustness of the aggregation. We instantiate our framework as RobAJoL, which integrates the Johnson-Lindenstrauss (JL)-based compression mechanism with robust averaging for robustness. Our theoretical analysis establishes the compatibility of JL transform with robust averaging, ensuring that RobAJoL maintains robustness guarantees, satisfies DP, and substantially reduces communication overhead. We further present simulation results on CIFAR-10, Fashion MNIST, and FEMNIST, validating our theoretical claims. We compare RobAJoL with a state-of-the-art communication-efficient and robust FL scheme augmented with DP for a fair comparison, demonstrating that RobAJoL outperforms existing methods in terms of robustness and utility under different Byzantine attacks.

Beyond Trade-offs: A Unified Framework for Privacy, Robustness, and Communication Efficiency in Federated Learning

TL;DR

The paper tackles the challenge of simultaneously ensuring privacy, robustness to Byzantine clients, and communication efficiency in federated learning. It introduces Fed-DPRoC, a framework that uses robust-compatible compression to preserve the guarantees of robust aggregation while reducing communication, and instantiates it as RobAJoL with JL-based compression and Gaussian DP. Theoretical results show that the JL transform preserves robustness with κ'=(1+ε_{JL})^2κ and δ_{RA}=ε_{JL}^2O(d)O(k)$. Empirical evaluations on CIFAR-10, Fashion-MNIST, and FEMNIST demonstrate that RobAJoL outperforms state-of-the-art DP+robust schemes across attacks and privacy budgets, achieving strong robustness and utility with significantly lowered communication overhead.

Abstract

We propose Fed-DPRoC, a novel federated learning framework designed to jointly provide differential privacy (DP), Byzantine robustness, and communication efficiency. Central to our approach is the concept of robust-compatible compression, which allows reducing the bi-directional communication overhead without undermining the robustness of the aggregation. We instantiate our framework as RobAJoL, which integrates the Johnson-Lindenstrauss (JL)-based compression mechanism with robust averaging for robustness. Our theoretical analysis establishes the compatibility of JL transform with robust averaging, ensuring that RobAJoL maintains robustness guarantees, satisfies DP, and substantially reduces communication overhead. We further present simulation results on CIFAR-10, Fashion MNIST, and FEMNIST, validating our theoretical claims. We compare RobAJoL with a state-of-the-art communication-efficient and robust FL scheme augmented with DP for a fair comparison, demonstrating that RobAJoL outperforms existing methods in terms of robustness and utility under different Byzantine attacks.

Paper Structure

This paper contains 24 sections, 8 theorems, 37 equations, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. System Model and Preliminaries
  3. Problem Formulation
  4. Preliminaries
  5. Robustness (Convergence)
  6. Differential Privacy
  7. The Proposed Framework: Fed-DPRoC
  8. RobAJoL: an Instantiation of Fed-DPRoC
  9. Johnson–Lindenstrauss (JL) Compression
  10. Robust Averaging and Compatibility of JL
  11. Privacy
  12. Theoretical Guarantees
  13. Experiments
  14. Training Parameters and Data Distribution
  15. Compression Method
  16. ...and 9 more sections

Key Result

Proposition 1

If a mechanism $f$ satisfies $(\alpha,\epsilon_\mathrm{RDP})$-RDP, then it also satisfies $(\epsilon_\mathrm{DP},\delta_\mathrm{{DP}})$-DP for any $0 < \delta_\mathrm{{DP}} < 1$, where $\epsilon_\mathrm{DP}=\epsilon_\mathrm{RDP}+\frac{\log (1/\delta_\mathrm{{DP}})}{\alpha-1}$.

Figures (5)

  • Figure 1: Illustration of Fed-DPRoC. Clients performs: $(1)$ minibatch sampling; $(2)$ decompression and model update; $(3)$ gradient computation; $(4)$ DP enhancement (\ref{['eq:dp_enforce']}); $(5)$ momentum computation (\ref{['eq:momentum']}); $(6)$ compression; and $(7)$ communication. Honest clients send their true compressed momenta to the federator, while malicious clients send arbitrarily manipulated vectors. The honest-but-curious federator: $(8)$ applies robust aggregation; and $(9)$ broadcasts the compressed aggregation.
  • Figure 2: A robust-compatible compression ensures that if $\textsc{Agg}\xspace$ is a robust aggregation rule, i.e., $\bm{m}_\textsc{Agg}\xspace$ is faithful to the average of the benign vectors $\bm{m}_1, \cdots, \bm{m}_n$, the composition $\textsc{Compress}\xspace \circ \textsc{Agg}\xspace \circ\textsc{Decompress}\xspace$ also gives an output that is faithful to the average of those benign vectors.
  • Figure 3: Test accuracy (unit: %) on non-i.i.d. CIFAR-10 under various attacks. We set $\sigma_\mathrm{{NM}}=0.1$ and compression rate $\frac{d}{k}=10$.
  • Figure 4: Test accuracy (unit: %) on FEMNIST under various attacks and different compression rates $1,10,100,500,1000$ and $2000$. We set the noise multiplier $\sigma_\mathrm{{NM}}=0.1$ and use Trimmed Mean as the robust aggregation rule.
  • Figure 5: Test accuracy (unit: %) on FEMNIST. We compare RobAJoL (on the left of each dotted line) with Byz-EF21-BC augmented with DP (on the right of each dotted line) under varying noise multipliers; Here, $\sigma_\mathrm{{NM}}=0$ indicates that no DP is applied. Both schemes use TM as the robust aggregation.

Theorems & Definitions (19)

  • Definition 1: $(b,\nu)$-robust algorithm guerraoui2024robust
  • Definition 2: $(\epsilon_\mathrm{DP},\delta_\mathrm{{DP}})$-Differential Privacy (DP) dwork2006our
  • Definition 3: R√©nyi Differential Privacy (RDP) mironov2017renyi
  • Proposition 1: RDP to DP conversion mironov2017renyi
  • Definition 4: Robust-compatible Compression
  • Definition 5: JL Lemma johnson1984extensionsdasgupta2003elementary
  • Example 1: Count-Sketch JL Transform chen2022fundamentalkane2014sparser
  • Lemma 1
  • Definition 6: $(b,\kappa)$-Robust Averaging guerraoui2024robust
  • Definition 7: $(b,\kappa,\delta_\text{RA})$-Robust Averaging
  • ...and 9 more