Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Learning with Differential Privacy: An Utility-Enhanced Approach

Kanishka Ranaweera, Dinh C. Nguyen, Pubudu N. Pathirana, David Smith, Ming Ding, Thierry Rakotoarivelo, Aruna Seneviratne

TL;DR

This work addresses privacy in federated learning by integrating a Haar wavelet transform into differentially private SGD and federated averaging. The core idea is to perform noise injection in the wavelet domain, with per-level clipping that reduces the effective noise variance and strengthens utility under the same $(\varepsilon,\delta)$ privacy budget. Theoretical results show improved noise variance bounds by a factor of $\sqrt{\dfrac{2+\log_2(m)}{2}}$, along with convergence guarantees, while empirical experiments on MNIST, Fashion-MNIST, and CIFAR-10 demonstrate superior model performance compared to standard DP baselines. The approach offers a practical path to higher-utility private FL, with future work exploring unequal noise across coefficients and adaptive clipping for non-i.i.d. data settings.

Abstract

Federated learning has emerged as an attractive approach to protect data privacy by eliminating the need for sharing clients' data while reducing communication costs compared with centralized machine learning algorithms. However, recent studies have shown that federated learning alone does not guarantee privacy, as private data may still be inferred from the uploaded parameters to the central server. In order to successfully avoid data leakage, adopting differential privacy (DP) in the local optimization process or in the local update aggregation process has emerged as two feasible ways for achieving sample-level or user-level privacy guarantees respectively, in federated learning models. However, compared to their non-private equivalents, these approaches suffer from a poor utility. To improve the privacy-utility trade-off, we present a modification to these vanilla differentially private algorithms based on a Haar wavelet transformation step and a novel noise injection scheme that significantly lowers the asymptotic bound of the noise variance. We also present a holistic convergence analysis of our proposed algorithm, showing that our method yields better convergence performance than the vanilla DP algorithms. Numerical experiments on real-world datasets demonstrate that our method outperforms existing approaches in model utility while maintaining the same privacy guarantees.

Federated Learning with Differential Privacy: An Utility-Enhanced Approach

TL;DR

This work addresses privacy in federated learning by integrating a Haar wavelet transform into differentially private SGD and federated averaging. The core idea is to perform noise injection in the wavelet domain, with per-level clipping that reduces the effective noise variance and strengthens utility under the same privacy budget. Theoretical results show improved noise variance bounds by a factor of , along with convergence guarantees, while empirical experiments on MNIST, Fashion-MNIST, and CIFAR-10 demonstrate superior model performance compared to standard DP baselines. The approach offers a practical path to higher-utility private FL, with future work exploring unequal noise across coefficients and adaptive clipping for non-i.i.d. data settings.

Abstract

Federated learning has emerged as an attractive approach to protect data privacy by eliminating the need for sharing clients' data while reducing communication costs compared with centralized machine learning algorithms. However, recent studies have shown that federated learning alone does not guarantee privacy, as private data may still be inferred from the uploaded parameters to the central server. In order to successfully avoid data leakage, adopting differential privacy (DP) in the local optimization process or in the local update aggregation process has emerged as two feasible ways for achieving sample-level or user-level privacy guarantees respectively, in federated learning models. However, compared to their non-private equivalents, these approaches suffer from a poor utility. To improve the privacy-utility trade-off, we present a modification to these vanilla differentially private algorithms based on a Haar wavelet transformation step and a novel noise injection scheme that significantly lowers the asymptotic bound of the noise variance. We also present a holistic convergence analysis of our proposed algorithm, showing that our method yields better convergence performance than the vanilla DP algorithms. Numerical experiments on real-world datasets demonstrate that our method outperforms existing approaches in model utility while maintaining the same privacy guarantees.

Paper Structure

This paper contains 22 sections, 3 theorems, 56 equations, 12 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Differential Privacy
  4. Federated Learning With Differential Privacy
  5. Scheme Overview
  6. System Model
  7. Threat Model
  8. Methodology
  9. DP-SGD via Wavelet Transforms
  10. DP-FedAvg via Wavelet Transforms
  11. One-dimensional Haar Wavelet Transform
  12. Theoretical analysis
  13. Noise Variance Bounds Analysis
  14. Differential Privacy Analysis
  15. Convergence Analysis of DP-SGD-WAV
  16. ...and 7 more sections

Key Result

Lemma 1

Assume that we perform HWT on a set of gradients $(G)$ with $m$ elements, which results in a set of wavelet coefficients $(H)$ where we add independent noise with standard deviation of $\dfrac{\sigma_{(Haar)}}{W_{(Haar)}}$. The set of noisy gradients ($G^*$) reconstructed from the inverse wavelet tr

Figures (12)

  • Figure 1: FL configurations with sample-level DP and user-level DP.
  • Figure 2: Example of a federated learning model.
  • Figure 3: Decomposition tree illustration of Haar wavelet transform.
  • Figure 4: Gaussian distribution curves with varying noise variance used in DP training. The red curve represents the original noise variance, while the blue curve represents the reduced noise variance proposed in our method. The x-axis shows the range of noise values, which is a real-valued axis that represents all possible noise values that can be added to the training process, and the y-axis represents the probability density.
  • Figure 5: Architecture of the Convolutional Neural Network (CNN) used to train on MNIST and Fashion MNIST datasets, consisting of multiple convolutional layers followed by pooling layers and fully connected layers.
  • ...and 7 more figures

Theorems & Definitions (8)

  • Definition 1
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Theorem 3
  • proof
  • proof