Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Observational Auditing of Label Privacy

Iden Kalemaj, Luca Melis, Maxime Boucher, Ilya Mironov, Saeed Mahloujifar

TL;DR

This work tackles the problem of auditing differential privacy in large-scale ML systems without altering the training data. It introduces an observational privacy auditing framework built on simulation-based DP, framing privacy as a post-training game between a simulator and an attacker. The authors specialize the framework to Label DP via a label-inference attack that uses counterfactual labels from a proxy model, deriving bounds that account for distribution shift. Empirical validation on CIFAR-10 and Criteo demonstrates practical, training-free privacy auditing with results aligning with interventional methods and offering a path for third-party privacy verification in production environments.

Abstract

Differential privacy (DP) auditing is essential for evaluating privacy guarantees in machine learning systems. Existing auditing methods, however, pose a significant challenge for large-scale systems since they require modifying the training dataset -- for instance, by injecting out-of-distribution canaries or removing samples from training. Such interventions on the training data pipeline are resource-intensive and involve considerable engineering overhead. We introduce a novel observational auditing framework that leverages the inherent randomness of data distributions, enabling privacy evaluation without altering the original dataset. Our approach extends privacy auditing beyond traditional membership inference to protected attributes, with labels as a special case, addressing a key gap in existing techniques. We provide theoretical foundations for our method and perform experiments on Criteo and CIFAR-10 datasets that demonstrate its effectiveness in auditing label privacy guarantees. This work opens new avenues for practical privacy auditing in large-scale production environments.

Observational Auditing of Label Privacy

TL;DR

This work tackles the problem of auditing differential privacy in large-scale ML systems without altering the training data. It introduces an observational privacy auditing framework built on simulation-based DP, framing privacy as a post-training game between a simulator and an attacker. The authors specialize the framework to Label DP via a label-inference attack that uses counterfactual labels from a proxy model, deriving bounds that account for distribution shift. Empirical validation on CIFAR-10 and Criteo demonstrates practical, training-free privacy auditing with results aligning with interventional methods and offering a path for third-party privacy verification in production environments.

Abstract

Differential privacy (DP) auditing is essential for evaluating privacy guarantees in machine learning systems. Existing auditing methods, however, pose a significant challenge for large-scale systems since they require modifying the training dataset -- for instance, by injecting out-of-distribution canaries or removing samples from training. Such interventions on the training data pipeline are resource-intensive and involve considerable engineering overhead. We introduce a novel observational auditing framework that leverages the inherent randomness of data distributions, enabling privacy evaluation without altering the original dataset. Our approach extends privacy auditing beyond traditional membership inference to protected attributes, with labels as a special case, addressing a key gap in existing techniques. We provide theoretical foundations for our method and perform experiments on Criteo and CIFAR-10 datasets that demonstrate its effectiveness in auditing label privacy guarantees. This work opens new avenues for practical privacy auditing in large-scale production environments.

Paper Structure

This paper contains 21 sections, 8 theorems, 47 equations, 5 figures, 7 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background and prior work
  3. Preliminaries
  4. Observational versus interventional privacy games
  5. Observational attribute inference
  6. Experiments on auditing Label DP
  7. Label DP learning algorithms
  8. Attack implementation and adversarial strategy
  9. CIFAR-10 experiments
  10. Criteo experiments
  11. Comparison with Existing Methods
  12. Synthetic data and auditing randomized response
  13. Discussion
  14. Simulation-based differential privacy
  15. Auditing --differential privacy
  16. ...and 6 more sections

Key Result

Theorem 5

Let $M\colon (\cX,\cY)^* \to \mathcal{Z}$ be a mechanism, $\cD$ the data distribution, $\cD'$ an approximate distribution, and $\mathrm{Sim}_{M, \cD'}$ the imputation-based simulator (Definition def:imputation-sim). Let $C = \sum_{i\in [m]} \mathbf{1}[b_i' = b_i]$ be the total number of correct answ If $M$ is $(\epsilon,\delta)$-SIM-DP with respect to $\mathrm{Sim}_{M, \cD'}$ and the auditing Algo

Figures (5)

  • Figure 1: Comparison with MIA for different Label DP Algorithms on CIFAR-10 and Criteo datasets. The error bar represents the standard deviation across 100 different repetitions.
  • Figure 2: Auditing randomized response when the adversary guesses on 0.1% of samples. The counterfactual labels are generated either from the ground-truth distribution or sampled according to the output of a logistic regression model.
  • Figure 3: Auditing randomized response when the adversary guesses on 1% of samples. The counterfactual labels are generated either from the ground-truth distribution or a proxy distribution yielded by the Logistic Regression model.
  • Figure 4: Auditing randomized response when synthetic labels are produced from a $\tau$-shifted distribution from the ground truth label distribution.
  • Figure 5: Comparison with calibration-based MIA watson2022difficulty and RMIA zarifzadeh2024low for different Label DP Algorithms on CIFAR-10 dataset. The error bar represents the standard deviation across 100 different attack repetitions.

Theorems & Definitions (21)

  • Definition 1: Simulation-based privacy for protected attributes mahloujifar2025machine
  • Definition 2: Imputation-based simulator
  • Definition 3: Auditing simulation-based DP
  • Definition 4: Observational games
  • Theorem 5: Auditing $f$-DP with distribution shift
  • Definition 6: Differential privacy
  • Definition : more formal version of Definition \ref{['def:sim-dp']}, Simulation-based privacy for protected attributes
  • Proposition 7: dong2019gaussian
  • Definition 8: Obtaining empirical epsilon from $f$-SIM-DP auditing
  • Theorem 9: Restated Theorem \ref{['thm:fdp-shift']}, auditing $f$-DP with distribution shift
  • ...and 11 more