The Effect of Quantization in Federated Learning: A Rényi Differential Privacy Perspective

TL;DR

This work analyzes how stochastic quantization affects privacy in Federated Learning when Gaussian noise is added for differential privacy. By deriving the privacy budget under Rényi Differential Privacy for the quantized Gaussian mechanism and showing that lower quantization levels tighten the budget, the authors reveal a beneficial privacy-communication trade-off. Theoretical results are corroborated with Membership Inference Attacks (MIA), demonstrating that quantization reduces privacy leakage at fixed noise and that leakage worsens as quantization becomes finer (larger $k$). The findings offer practical guidance for designing communication-efficient, privacy-preserving FL systems using quantization, with quantified guarantees and empirical validation on standard benchmarks like CIFAR-10. Overall, the paper connects quantization, DP, and FL performance to illuminate how memory- and bandwidth-efficient quantization can reinforce privacy protections without sacrificing interpretability of the privacy budget.

Abstract

Federated Learning (FL) is an emerging paradigm that holds great promise for privacy-preserving machine learning using distributed data. To enhance privacy, FL can be combined with Differential Privacy (DP), which involves adding Gaussian noise to the model weights. However, FL faces a significant challenge in terms of large communication overhead when transmitting these model weights. To address this issue, quantization is commonly employed. Nevertheless, the presence of quantized Gaussian noise introduces complexities in understanding privacy protection. This research paper investigates the impact of quantization on privacy in FL systems. We examine the privacy guarantees of quantized Gaussian mechanisms using Rényi Differential Privacy (RDP). By deriving the privacy budget of quantized Gaussian mechanisms, we demonstrate that lower quantization bit levels provide improved privacy protection. To validate our theoretical findings, we employ Membership Inference Attacks (MIA), which gauge the accuracy of privacy leakage. The numerical results align with our theoretical analysis, confirming that quantization can indeed enhance privacy protection. This study not only enhances our understanding of the correlation between privacy and communication in FL but also underscores the advantages of quantization in preserving privacy.

Figures (5)

• Figure 1: Illustration of Federated Learning with quantized Gaussian mechanism applied. The weight of each client is first perturbed by Gaussian noise and then passed to the quantizer.
• Figure 2: Working flow of the offline variant of LiRA. The attacker trains a set of shadow models that do not contain $x$ in their training set, and utilizes them to estimate $\mu_\text{out}$, $\sigma_\text{out}$. The attacker uses $\Pr\left[l | N_\text{out}\right]$ to infer membership of $x$.
• Figure 3: Comparison of RDP Privacy budgets for Gaussian and quantized Gaussian mechanisms at $\alpha=1$, with varying quantization levels $k$ from 2 to 64 while keeping $\sigma=1$ and $\Delta_2=1$. $x$-axis depicted in a logarithmic scale represents the quantization level $k$.
• Figure 4: Comparison of utility and MIA accuracy among models employing Gaussian and quantized Gaussian mechanism, all with a noise variance of 0.03. The dashed line represents the model without quantization. $x$-axis depicted in a logarithmic scale represents the quantization level $k$.
• Figure 5: Utility and MIA accuracy comparison among models employing Gaussian mechanism and quantized Gaussian mechanism with quantization level $k=16$ and $k=32$, varying the Gaussian noise variance during training.

Theorems & Definitions (10)

• Definition 1: $k$-Level Stochastic Quantizer with clipping
• Definition 2: $(\epsilon, \delta)$-DP
• Definition 3: $(\alpha, \epsilon)$-RDP and $\ell_2$-sensitivity
• Lemma 1: monotonicity
• Theorem 1: Post-processing dwork2014algorithmic
• Lemma 2
• Theorem 2
• proof
• Lemma 3
• proof