Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Communication-Efficient and Privacy-Adaptable Mechanism for Federated Learning

Chih Wei Ling, Chun Hei Michael Shiu, Youqi Wu, Jiande Sun, Cheuk Ting Li, Linqi Song, Weitao Xu

TL;DR

CEPAM tackles communication efficiency and privacy in Federated Learning by using rejection-sampled universal quantization (RSUQ) to turn quantization distortion into controlled additive noise, enabling joint DP and compression with privacy-adaptability. It extends RSUQ into layered RSUQ (LRSUQ) to simulate Gaussian or Laplace noise, providing CEPAM-Gaussian and CEPAM-Laplace variants with formal DP guarantees and privacy amplification analysis. The framework is validated on MNIST with MLP and CNN models, showing consistent accuracy gains over baselines and improved compression relative to scalar quantization. These results demonstrate a practical, adaptable approach for privacy-preserving FL in bandwidth-constrained environments, with clear pathways to broader noise distributions and non-convex objectives.

Abstract

Training machine learning models on decentralized private data via federated learning (FL) poses two key challenges: communication efficiency and privacy protection. In this work, we address these challenges within the trusted aggregator model by introducing a novel approach called the Communication-Efficient and Privacy-Adaptable Mechanism (CEPAM), achieving both objectives simultaneously. In particular, CEPAM leverages the rejection-sampled universal quantizer (RSUQ), a construction of randomized vector quantizer whose resulting distortion is equivalent to a prescribed noise, such as Gaussian or Laplace noise, enabling joint differential privacy and compression. Our CEPAM provides the additional benefit of privacy adaptability, allowing clients and the server to customize privacy protection based on required accuracy and protection. We theoretically analyze the privacy guarantee of CEPAM and investigate the trade-offs among user privacy and accuracy of CEPAM through experimental evaluations. Moreover, we assess CEPAM's utility performance using MNIST dataset, demonstrating that CEPAM surpasses baseline models in terms of learning accuracy.

Communication-Efficient and Privacy-Adaptable Mechanism for Federated Learning

TL;DR

CEPAM tackles communication efficiency and privacy in Federated Learning by using rejection-sampled universal quantization (RSUQ) to turn quantization distortion into controlled additive noise, enabling joint DP and compression with privacy-adaptability. It extends RSUQ into layered RSUQ (LRSUQ) to simulate Gaussian or Laplace noise, providing CEPAM-Gaussian and CEPAM-Laplace variants with formal DP guarantees and privacy amplification analysis. The framework is validated on MNIST with MLP and CNN models, showing consistent accuracy gains over baselines and improved compression relative to scalar quantization. These results demonstrate a practical, adaptable approach for privacy-preserving FL in bandwidth-constrained environments, with clear pathways to broader noise distributions and non-convex objectives.

Abstract

Training machine learning models on decentralized private data via federated learning (FL) poses two key challenges: communication efficiency and privacy protection. In this work, we address these challenges within the trusted aggregator model by introducing a novel approach called the Communication-Efficient and Privacy-Adaptable Mechanism (CEPAM), achieving both objectives simultaneously. In particular, CEPAM leverages the rejection-sampled universal quantizer (RSUQ), a construction of randomized vector quantizer whose resulting distortion is equivalent to a prescribed noise, such as Gaussian or Laplace noise, enabling joint differential privacy and compression. Our CEPAM provides the additional benefit of privacy adaptability, allowing clients and the server to customize privacy protection based on required accuracy and protection. We theoretically analyze the privacy guarantee of CEPAM and investigate the trade-offs among user privacy and accuracy of CEPAM through experimental evaluations. Moreover, we assess CEPAM's utility performance using MNIST dataset, demonstrating that CEPAM surpasses baseline models in terms of learning accuracy.
Paper Structure (30 sections, 11 theorems, 21 equations, 3 figures, 2 tables, 3 algorithms)

This paper contains 30 sections, 11 theorems, 21 equations, 3 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction and Motivation
  2. Related Works
  3. System Model and Preliminaries
  4. Federated Learning (FL)
  5. Rejection-Sampled Universal Quantizer (RSUQ)
  6. Differential Privacy (DP)
  7. Problem Setting
  8. Threat model
  9. CEPAM for FL
  10. Performance Analysis
  11. Privacy
  12. Gaussian mechanism
  13. Laplace mechanism
  14. Compression
  15. Numerical Evaluations
  16. ...and 15 more sections

Key Result

Proposition 3

Ling2024arxivRSUQ For any $\mathbf{x} \in \mathbb{R}^n$, the quantization error $\mathbf{Z} := Q_{\mathcal{A},\mathcal{P}}(\mathbf{x},S) - \mathbf{x}$ of RSUQ $Q_{\mathcal{A},\mathcal{P}}$, where $S \sim P_S$, follows the uniform distribution over the set $\mathcal{A}$, i.e., $\mathbf{Z} \sim \mathr

Figures (3)

  • Figure 1: Convergence profile of different FL schemes for CEPAM-Gaussian.
  • Figure 2: Convergence profiles of different FL schemes for CEPAM-Laplace.
  • Figure 3: Learning Accuracy and Privacy Trade-off using CNN

Theorems & Definitions (16)

  • Definition 1
  • Definition 2
  • Proposition 3
  • Definition 4
  • Proposition 5
  • Definition 6: $(\epsilon, \delta)$-differential privacy Dwork06DP
  • Theorem 7
  • Theorem 8
  • Lemma 9
  • proof
  • ...and 6 more