Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributed, communication-efficient, and differentially private estimation of KL divergence

Mary Scott, Sayan Biswas, Graham Cormode, Carsten Maple

TL;DR

This paper addresses measuring distribution drift in privacy-sensitive federated data by estimating the KL divergence $D_{KL}(\\Pi \\| P)$ between a reference distribution and a private data distribution. It introduces PRIEST-KLD, a family of three DP-enabled estimators that trade off trust, communication, and accuracy across Fully Trusted, TAgg, and Fully Distributed models, using Monte Carlo subsampling and secure aggregation. The estimators are shown to be unbiased in the Trusted and TAgg models, with DP noise added to guarantee $(\\varepsilon,\\delta)$-DP, while the Dist model relies on local DP noise and yields a non-unbiased estimator but strong privacy and low communication. Empirical results on FEMNIST demonstrate that the Dist model often achieves the best privacy-utility trade-off, guiding parameter choices such as the noise level via $\\varepsilon$, the sampling rate via $|C_t|$, and the variance-control parameter via $\\lambda$ for practical deployment.

Abstract

A key task in managing distributed, sensitive data is to measure the extent to which a distribution changes. Understanding this drift can effectively support a variety of federated learning and analytics tasks. However, in many practical settings sharing such information can be undesirable (e.g., for privacy concerns) or infeasible (e.g., for high communication costs). In this work, we describe novel algorithmic approaches for estimating the KL divergence of data across federated models of computation, under differential privacy. We analyze their theoretical properties and present an empirical study of their performance. We explore parameter settings that optimize the accuracy of the algorithm catering to each of the settings; these provide sub-variations that are applicable to real-world tasks, addressing different context- and application-specific trust level requirements. Our experimental results confirm that our private estimators achieve accuracy comparable to a baseline algorithm without differential privacy guarantees.

Distributed, communication-efficient, and differentially private estimation of KL divergence

TL;DR

This paper addresses measuring distribution drift in privacy-sensitive federated data by estimating the KL divergence between a reference distribution and a private data distribution. It introduces PRIEST-KLD, a family of three DP-enabled estimators that trade off trust, communication, and accuracy across Fully Trusted, TAgg, and Fully Distributed models, using Monte Carlo subsampling and secure aggregation. The estimators are shown to be unbiased in the Trusted and TAgg models, with DP noise added to guarantee -DP, while the Dist model relies on local DP noise and yields a non-unbiased estimator but strong privacy and low communication. Empirical results on FEMNIST demonstrate that the Dist model often achieves the best privacy-utility trade-off, guiding parameter choices such as the noise level via , the sampling rate via , and the variance-control parameter via for practical deployment.

Abstract

A key task in managing distributed, sensitive data is to measure the extent to which a distribution changes. Understanding this drift can effectively support a variety of federated learning and analytics tasks. However, in many practical settings sharing such information can be undesirable (e.g., for privacy concerns) or infeasible (e.g., for high communication costs). In this work, we describe novel algorithmic approaches for estimating the KL divergence of data across federated models of computation, under differential privacy. We analyze their theoretical properties and present an empirical study of their performance. We explore parameter settings that optimize the accuracy of the algorithm catering to each of the settings; these provide sub-variations that are applicable to real-world tasks, addressing different context- and application-specific trust level requirements. Our experimental results confirm that our private estimators achieve accuracy comparable to a baseline algorithm without differential privacy guarantees.

Paper Structure

This paper contains 26 sections, 10 theorems, 26 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries and Background
  4. Probability Distributions and Divergences
  5. Federated Computation
  6. Privacy Standards
  7. Monte Carlo subsampling
  8. Differentially private estimator of KL divergence
  9. Problem definition
  10. PRIEST-KLD: Trusted models
  11. Fully trusted federated model
  12. PRIEST-KLD : Trusted aggregator (TAgg) model
  13. Privacy analysis
  14. PRIEST-KLD: Fully distributed model
  15. Experimental evaluation
  16. ...and 11 more sections

Key Result

Corollary 3.1

For any $\varepsilon, \delta \geq 0$, it suffices to sample Gaussian noise $\eta_G \sim \mathcal{N}\space\!\left( 0, \frac{ 2 \log \frac{1.25}{\delta} \Delta(f)}{\varepsilon} \right)$ and add this to $f(P)$ to provide $(\varepsilon, \delta)$-DP for $f$.

Figures (5)

  • Figure 1: Effect of $\lambda$ on mean MSE of PRIEST-KLD models, fixing $|C_{t}\space| \space = 200$.
  • Figure 2: Effect of $\lambda$ on mean MSE of PRIEST-KLD models, fixing $\varepsilon = 0.5$.
  • Figure 3: Effect of $\varepsilon$ on MSE of PRIEST-KLD models, fixing $|C_{t}\space| \space = 200$.
  • Figure 4: Effect of $\varepsilon$ on percentage of $D_{KL}$ that is noise, fixing $|C_{t}\space| \space = 200$.
  • Figure 5: Effect of $|C_{t}\space|$ on MSE of PRIEST-KLD models, fixing $\varepsilon = 0.5$, $\varepsilon = 2$ or $\varepsilon = 5$.

Theorems & Definitions (28)

  • Definition 3.1
  • Definition 3.2: $(\varepsilon, \delta)$-DP dwork
  • Definition 3.3: Adjacent histograms
  • Definition 3.4: Adjacent distributions
  • Definition 3.5: Sensitivity
  • Corollary 3.1: Gaussian noise dwork
  • proof
  • proof
  • Definition 4.1: Optimal variance parameter
  • Theorem 4.3
  • ...and 18 more