Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Auditing Private Prediction

Karan Chadha, Matthew Jagielski, Nicolas Papernot, Christopher Choquette-Choo, Milad Nasr

TL;DR

This work introduces the first auditing framework for private prediction, extending differential privacy auditing to inference-time privacy leakage. By modeling per-query Renyi DP and employing exact Renyi divergence calculations for noisy-argmax across histograms, the authors derive informative lower and upper bounds and apply them to four private-prediction algorithms: PATE, CaPC, PromptPATE, and Private-kNN. They design adversaries with varying poisoning and query capabilities (Nat-MIAQ, Pois-MIAQ, Pois-MINQ) and demonstrate that leakage grows with poisoning ability and query control, while data-dependent analyses can tighten or loosen bounds. The framework enables attribution of leakage to algorithm design and adversarial capabilities, offering a practical tool for assessing privacy of private-prediction systems in real-world tasks such as LLM task adaptation. The results indicate substantial room to improve privacy analyses and suggest directions for stronger attacks, distribution-aware auditing, and leveraging partitioning randomness to refine bounds.

Abstract

Differential privacy (DP) offers a theoretical upper bound on the potential privacy leakage of analgorithm, while empirical auditing establishes a practical lower bound. Auditing techniques exist forDP training algorithms. However machine learning can also be made private at inference. We propose thefirst framework for auditing private prediction where we instantiate adversaries with varying poisoningand query capabilities. This enables us to study the privacy leakage of four private prediction algorithms:PATE [Papernot et al., 2016], CaPC [Choquette-Choo et al., 2020], PromptPATE [Duan et al., 2023],and Private-kNN [Zhu et al., 2020]. To conduct our audit, we introduce novel techniques to empiricallyevaluate privacy leakage in terms of Renyi DP. Our experiments show that (i) the privacy analysis ofprivate prediction can be improved, (ii) algorithms which are easier to poison lead to much higher privacyleakage, and (iii) the privacy leakage is significantly lower for adversaries without query control than thosewith full control.

Auditing Private Prediction

TL;DR

This work introduces the first auditing framework for private prediction, extending differential privacy auditing to inference-time privacy leakage. By modeling per-query Renyi DP and employing exact Renyi divergence calculations for noisy-argmax across histograms, the authors derive informative lower and upper bounds and apply them to four private-prediction algorithms: PATE, CaPC, PromptPATE, and Private-kNN. They design adversaries with varying poisoning and query capabilities (Nat-MIAQ, Pois-MIAQ, Pois-MINQ) and demonstrate that leakage grows with poisoning ability and query control, while data-dependent analyses can tighten or loosen bounds. The framework enables attribution of leakage to algorithm design and adversarial capabilities, offering a practical tool for assessing privacy of private-prediction systems in real-world tasks such as LLM task adaptation. The results indicate substantial room to improve privacy analyses and suggest directions for stronger attacks, distribution-aware auditing, and leveraging partitioning randomness to refine bounds.

Abstract

Differential privacy (DP) offers a theoretical upper bound on the potential privacy leakage of analgorithm, while empirical auditing establishes a practical lower bound. Auditing techniques exist forDP training algorithms. However machine learning can also be made private at inference. We propose thefirst framework for auditing private prediction where we instantiate adversaries with varying poisoningand query capabilities. This enables us to study the privacy leakage of four private prediction algorithms:PATE [Papernot et al., 2016], CaPC [Choquette-Choo et al., 2020], PromptPATE [Duan et al., 2023],and Private-kNN [Zhu et al., 2020]. To conduct our audit, we introduce novel techniques to empiricallyevaluate privacy leakage in terms of Renyi DP. Our experiments show that (i) the privacy analysis ofprivate prediction can be improved, (ii) algorithms which are easier to poison lead to much higher privacyleakage, and (iii) the privacy leakage is significantly lower for adversaries without query control than thosewith full control.
Paper Structure (56 sections, 1 theorem, 16 equations, 31 figures, 2 tables, 1 algorithm)

This paper contains 56 sections, 1 theorem, 16 equations, 31 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Notation and Preliminaries
  4. Differential Privacy
  5. Auditing Differential Privacy
  6. Attack Interpretation.
  7. Private Prediction
  8. Data Independent.
  9. Data Dependent.
  10. Auditing Private Prediction
  11. Adversaries
  12. $\textbf{Nat-MIAQ}$.
  13. $\textbf{Pois-MIAQ}$.
  14. $\textbf{Pois-MINQ}$.
  15. Auditing RDP of Noisy Argmax
  16. ...and 41 more sections

Key Result

Theorem 1

balle2020hypothesis If an algorithm is $(\alpha,\varepsilon_{\alpha})$-RDP, then it is also $(\varepsilon_{\alpha} + \log(\frac{\alpha - 1}{\alpha}) - \frac{\log{\delta} + \log{\alpha}}{\alpha - 1},\delta)$-DP for any $\delta \in (0,1)$.

Figures (31)

  • Figure 1: Framework to audit private prediction algorithms.
  • Figure 2: RDP audit for noisy argmax
  • Figure 3: Privacy leakage of PATE, CaPC and PromptPATE for adversaries with adversarial query capability.
  • Figure 4: Privacy leakage of PATE, CaPC and PromptPATE for adversaries restricted to making natural queries.
  • Figure 5: Privacy leakage for Private-kNN
  • ...and 26 more figures

Theorems & Definitions (4)

  • Definition 1: Approximate DP
  • Definition 2: Renyi DP
  • Theorem 1
  • Definition 3: Private Prediction Interface