Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PAC-Private Responses with Adversarial Composition

Xiaochen Zhu, Mayuri Sridhar, Srinivas Devadas

TL;DR

The paper introduces PAC-private responses to privatize ML inference in API-like settings, addressing the challenge of adversarial adaptive querying. By maintaining a Bayesian posterior of the secret and adaptively calibrating Gaussian noise, the authors prove a linear mutual information composition bound I(S;R1,...,RT) ≤ ∑t bt under adversarial interaction, enabling scalable privacy with high utility. The approach is instantiated for ML predictions, including a practical mechanism with offline pretraining over a finite secret set, outputting privatized hard labels, and a confidence-filtered private distillation pipeline that yields strong performance on CIFAR-10 and CIFAR-100 while bounding membership inference attacks. Extensive experiments across tabular, vision, and NLP tasks demonstrate competitive accuracy and the ability to serve millions of queries under tight privacy budgets, with distilled models achieving near DP-like guarantees. The work offers a practical, model-agnostic privacy framework for private responses and points to future directions in extending the theory and applying the paradigm to large-language models and broader iterative algorithms.

Abstract

Modern machine learning models are increasingly deployed behind APIs. This renders standard weight-privatization methods (e.g. DP-SGD) unnecessarily noisy at the cost of utility. While model weights may vary significantly across training datasets, model responses to specific inputs are much lower dimensional and more stable. This motivates enforcing privacy guarantees directly on model outputs. We approach this under PAC privacy, which provides instance-based privacy guarantees for arbitrary black-box functions by controlling mutual information (MI). Importantly, PAC privacy explicitly rewards output stability with reduced noise levels. However, a central challenge remains: response privacy requires composing a large number of adaptively chosen, potentially adversarial queries issued by untrusted users, where existing composition results on PAC privacy are inadequate. We introduce a new algorithm that achieves adversarial composition via adaptive noise calibration and prove that mutual information guarantees accumulate linearly under adaptive and adversarial querying. Experiments across tabular, vision, and NLP tasks show that our method achieves high utility at extremely small per-query privacy budgets. On CIFAR-10, we achieve 87.79% accuracy with a per-step MI budget of $2^{-32}$. This enables serving one million queries while provably bounding membership inference attack (MIA) success rates to 51.08% -- the same guarantee of $(0.04, 10^{-5})$-DP. Furthermore, we show that private responses can be used to label public data to distill a publishable privacy-preserving model; using an ImageNet subset as a public dataset, our model distilled from 210,000 responses achieves 91.86% accuracy on CIFAR-10 with MIA success upper-bounded by 50.49%, which is comparable to $(0.02,10^{-5})$-DP.

PAC-Private Responses with Adversarial Composition

TL;DR

The paper introduces PAC-private responses to privatize ML inference in API-like settings, addressing the challenge of adversarial adaptive querying. By maintaining a Bayesian posterior of the secret and adaptively calibrating Gaussian noise, the authors prove a linear mutual information composition bound I(S;R1,...,RT) ≤ ∑t bt under adversarial interaction, enabling scalable privacy with high utility. The approach is instantiated for ML predictions, including a practical mechanism with offline pretraining over a finite secret set, outputting privatized hard labels, and a confidence-filtered private distillation pipeline that yields strong performance on CIFAR-10 and CIFAR-100 while bounding membership inference attacks. Extensive experiments across tabular, vision, and NLP tasks demonstrate competitive accuracy and the ability to serve millions of queries under tight privacy budgets, with distilled models achieving near DP-like guarantees. The work offers a practical, model-agnostic privacy framework for private responses and points to future directions in extending the theory and applying the paradigm to large-language models and broader iterative algorithms.

Abstract

Modern machine learning models are increasingly deployed behind APIs. This renders standard weight-privatization methods (e.g. DP-SGD) unnecessarily noisy at the cost of utility. While model weights may vary significantly across training datasets, model responses to specific inputs are much lower dimensional and more stable. This motivates enforcing privacy guarantees directly on model outputs. We approach this under PAC privacy, which provides instance-based privacy guarantees for arbitrary black-box functions by controlling mutual information (MI). Importantly, PAC privacy explicitly rewards output stability with reduced noise levels. However, a central challenge remains: response privacy requires composing a large number of adaptively chosen, potentially adversarial queries issued by untrusted users, where existing composition results on PAC privacy are inadequate. We introduce a new algorithm that achieves adversarial composition via adaptive noise calibration and prove that mutual information guarantees accumulate linearly under adaptive and adversarial querying. Experiments across tabular, vision, and NLP tasks show that our method achieves high utility at extremely small per-query privacy budgets. On CIFAR-10, we achieve 87.79% accuracy with a per-step MI budget of . This enables serving one million queries while provably bounding membership inference attack (MIA) success rates to 51.08% -- the same guarantee of -DP. Furthermore, we show that private responses can be used to label public data to distill a publishable privacy-preserving model; using an ImageNet subset as a public dataset, our model distilled from 210,000 responses achieves 91.86% accuracy on CIFAR-10 with MIA success upper-bounded by 50.49%, which is comparable to -DP.
Paper Structure (45 sections, 7 theorems, 19 equations, 3 figures, 6 tables, 1 algorithm)

This paper contains 45 sections, 7 theorems, 19 equations, 3 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Our Contributions
  3. Paper Organization
  4. Background
  5. PAC Privacy
  6. Membership Inference Attacks
  7. Adversarial Composition
  8. Setting
  9. Posterior-Aware Composition
  10. Privacy Guarantee
  11. Computational Complexity
  12. PAC-Private Machine Learning Responses
  13. Setting and Threat Model
  14. Threat Model
  15. Concrete Instantiation
  16. ...and 30 more sections

Key Result

theorem 2

For any adversary $A$, its posterior advantage satisfies $\Delta_f\delta_{A}\leq \inf_{P_W}D_f(P_{S,M(S)}\,\|\, P_S\otimes P_W)$, where $P_W$ is the distribution of an arbitrary random variable $W$ with support $\mathcal{R}$. In particular, when $D_f$ is the KL-divergence and $P_W=P_{M(S)}$, we have where $I(\cdotp;\cdot)$ is the mutual information between two random variables.

Figures (3)

  • Figure 1: Provable upper bound on the individual MIA success rate as a function of the MI budget B, under a uniform prior success rate of 50%. Differential privacy parameters with matching guarantees are showed in brackets for reference.
  • Figure 2: An informed adversary refines their knowledge of the secret by tracking the posterior distribution of the secret conditioned on the observed transcript. In response, the curator adaptively calibrates noise to bound the conditional MI given the current transcript, i.e., the expected KL divergence between the posterior distribution and the prior $P_S$.
  • Figure 3: An overview of our proposed system to release PAC-private machine learning responses to an adversary's queries.

Theorems & Definitions (15)

  • definition 1: $(\delta,\rho,P_S)$-PAC Privacy hanshen2023crypto
  • Remark 1
  • definition 2: Posterior Advantage hanshen2023crypto
  • theorem 2: hanshen2023crypto
  • theorem 3: Noise Determination hanshen2023cryptomayuri2025sp
  • Remark 4
  • definition 3: Membership Inference Attack mayuri2025sp
  • definition 4: Noise Calibration Function $\Sigma$
  • lemma 1: Validity of Posterior Update
  • theorem 5: Adversarial Composition
  • ...and 5 more