Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning from End User Data with Shuffled Differential Privacy over Kernel Densities

Tal Wagner

TL;DR

This paper studies private data collection from end users under the shuffled differential privacy model and proposes a KDE-based private learning framework that scales to multi-class classification. The authors develop a kernel density estimation protocol in the shuffled DP setting by reducing KDE to bitsum computations via locality-sensitive quantization, achieving an $(\varepsilon,\delta)$-DP guarantee with supRMSE close to the central DP benchmark. They then apply per-class KDEs to perform private classification using a highest-density class rule and introduce private class decoding to extract semantic content from learned class representations, even without unprotected class data. Experiments on textual and visual embeddings demonstrate favorable downstream performance, revealing important privacy-accuracy-communication trade-offs in practical shuffled-DP ML deployments.

Abstract

We study a setting of collecting and learning from private data distributed across end users. In the shuffled model of differential privacy, the end users partially protect their data locally before sharing it, and their data is also anonymized during its collection to enhance privacy. This model has recently become a prominent alternative to central DP, which requires full trust in a central data curator, and local DP, where fully local data protection takes a steep toll on downstream accuracy. Our main technical result is a shuffled DP protocol for privately estimating the kernel density function of a distributed dataset, with accuracy essentially matching central DP. We use it to privately learn a classifier from the end user data, by learning a private density function per class. Moreover, we show that the density function itself can recover the semantic content of its class, despite having been learned in the absence of any unprotected data. Our experiments show the favorable downstream performance of our approach, and highlight key downstream considerations and trade-offs in a practical ML deployment of shuffled DP.

Learning from End User Data with Shuffled Differential Privacy over Kernel Densities

TL;DR

This paper studies private data collection from end users under the shuffled differential privacy model and proposes a KDE-based private learning framework that scales to multi-class classification. The authors develop a kernel density estimation protocol in the shuffled DP setting by reducing KDE to bitsum computations via locality-sensitive quantization, achieving an -DP guarantee with supRMSE close to the central DP benchmark. They then apply per-class KDEs to perform private classification using a highest-density class rule and introduce private class decoding to extract semantic content from learned class representations, even without unprotected class data. Experiments on textual and visual embeddings demonstrate favorable downstream performance, revealing important privacy-accuracy-communication trade-offs in practical shuffled-DP ML deployments.

Abstract

We study a setting of collecting and learning from private data distributed across end users. In the shuffled model of differential privacy, the end users partially protect their data locally before sharing it, and their data is also anonymized during its collection to enhance privacy. This model has recently become a prominent alternative to central DP, which requires full trust in a central data curator, and local DP, where fully local data protection takes a steep toll on downstream accuracy. Our main technical result is a shuffled DP protocol for privately estimating the kernel density function of a distributed dataset, with accuracy essentially matching central DP. We use it to privately learn a classifier from the end user data, by learning a private density function per class. Moreover, we show that the density function itself can recover the semantic content of its class, despite having been learned in the absence of any unprotected data. Our experiments show the favorable downstream performance of our approach, and highlight key downstream considerations and trade-offs in a practical ML deployment of shuffled DP.

Paper Structure

This paper contains 34 sections, 5 theorems, 23 equations, 8 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Preliminaries
  3. Central, Local and Shuffled DP
  4. Bitsum Protocols in the Shuffled DP Model
  5. Private Kernel Density Estimation
  6. Additional Related Work
  7. Data Collection and Classification with Shuffled DP
  8. Practical Considerations with Shuffled DP
  9. Shuffled DP KDE from Bitsum Protocols
  10. Private Learning, Classification and Class Decoding
  11. Experiments
  12. Private Classification Results
  13. Communication Cost
  14. Private KDE results
  15. Private Class Decoding Results
  16. ...and 19 more sections

Key Result

Theorem 3.1

Let $\mathrm{\mathbf{k}}$ be a $\beta$-approximate $(Q,R,S)$-LSQable kernel (cf. def:lsq). Suppose we have an unbiased $(\varepsilon_0,\delta_0)$-DP bitsum protocol $\Pi$ in the shuffled DP model, with RMSE $\mathcal{E}_{\Pi}$. Then, for every $\delta'>0$ and integer $I>0$, Algorithm alg:shufdpkde i

Figures (8)

  • Figure 1: Classification results with $\varepsilon_{\mathrm{lbl}}=5$
  • Figure 2: Empirical communication
  • Figure 3: Gaussian KDE accuracy
  • Figure 4: Classification accuracy comparison with a local DP baseline (overlaid on the shuffled DP RR plots with $\varepsilon_{\mathrm{lbl}}=5$, from the leftmost column in Figure \ref{['fig:epslbl5_primary']}).
  • Figure 5: Classification results with $\varepsilon_{\mathrm{lbl}}=10$
  • ...and 3 more figures

Theorems & Definitions (14)

  • Definition 1: wagner2023fast
  • Definition 2: shuffled DP KDE
  • Theorem 3.1
  • Theorem 3.2: shuffled DP Gaussian KDE
  • Corollary 1
  • Theorem A.1: \ref{['thm:main']}, restated
  • Claim 1
  • proof
  • Claim 2
  • proof
  • ...and 4 more