Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentially private ratio statistics

Tomer Shoham, Katrina Ligettt

TL;DR

The paper investigates differentially private ratio statistics, focusing on the relative risk, and shows that simple post-processing of private counts can yield strong privacy, accuracy, and bias properties even at small sample sizes. It analyzes a DP estimator for relative risk, proves consistency, and develops valid confidence intervals that combine sampling and privacy noise. By deriving finite-sample guarantees and providing extensive numerical studies, the work offers practical guidance for implementing private ratio estimation in real pipelines. The results bridge a gap in the DP literature by showing that ratio statistics can remain informative under privacy constraints, with manageable loss in accuracy in typical regimes and robust CI methods for finite samples. This has practical implications for private hypothesis testing, model evaluation, and fairness analyses in machine learning.

Abstract

Ratio statistics--such as relative risk and odds ratios--play a central role in hypothesis testing, model evaluation, and decision-making across many areas of machine learning, including causal inference and fairness analysis. However, despite privacy concerns surrounding many datasets and despite increasing adoption of differential privacy, differentially private ratio statistics have largely been neglected by the literature and have only recently received an initial treatment by Lin et al. [1]. This paper attempts to fill this lacuna, giving results that can guide practice in evaluating ratios when the results must be protected by differential privacy. In particular, we show that even a simple algorithm can provide excellent properties concerning privacy, sample accuracy, and bias, not just asymptotically but also at quite small sample sizes. Additionally, we analyze a differentially private estimator for relative risk, prove its consistency, and develop a method for constructing valid confidence intervals. Our approach bridges a gap in the differential privacy literature and provides a practical solution for ratio estimation in private machine learning pipelines.

Differentially private ratio statistics

TL;DR

The paper investigates differentially private ratio statistics, focusing on the relative risk, and shows that simple post-processing of private counts can yield strong privacy, accuracy, and bias properties even at small sample sizes. It analyzes a DP estimator for relative risk, proves consistency, and develops valid confidence intervals that combine sampling and privacy noise. By deriving finite-sample guarantees and providing extensive numerical studies, the work offers practical guidance for implementing private ratio estimation in real pipelines. The results bridge a gap in the DP literature by showing that ratio statistics can remain informative under privacy constraints, with manageable loss in accuracy in typical regimes and robust CI methods for finite samples. This has practical implications for private hypothesis testing, model evaluation, and fairness analyses in machine learning.

Abstract

Ratio statistics--such as relative risk and odds ratios--play a central role in hypothesis testing, model evaluation, and decision-making across many areas of machine learning, including causal inference and fairness analysis. However, despite privacy concerns surrounding many datasets and despite increasing adoption of differential privacy, differentially private ratio statistics have largely been neglected by the literature and have only recently received an initial treatment by Lin et al. [1]. This paper attempts to fill this lacuna, giving results that can guide practice in evaluating ratios when the results must be protected by differential privacy. In particular, we show that even a simple algorithm can provide excellent properties concerning privacy, sample accuracy, and bias, not just asymptotically but also at quite small sample sizes. Additionally, we analyze a differentially private estimator for relative risk, prove its consistency, and develop a method for constructing valid confidence intervals. Our approach bridges a gap in the differential privacy literature and provides a practical solution for ratio estimation in private machine learning pipelines.

Paper Structure

This paper contains 37 sections, 14 theorems, 79 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminaries
  4. A ratio of two counts
  5. Differential privacy
  6. Properties of the statistic
  7. Other methods for DP-ratio estimation
  8. Direct perturbation
  9. Local-sensitivity-based methods
  10. A ratio of noisy counts
  11. Unprocessed ratio of noisy counts
  12. Quantifying and correcting the bias
  13. Consistency and confidence interval of the relative risk statistic
  14. Confidence interval for the relative risk statistic
  15. A more conservative CI for small sample sizes
  16. ...and 22 more sections

Key Result

Corollary 3.3

Fix some $Z \in {\mathbb R}^+$. Consider $Y\in {\mathbb R}^+$ and let $X$ be $X=ZY$. For any $\varepsilon>0$ and $\alpha$ such that $0<\alpha<Z$, if we denote the output of Algorithm LaplaceNoisedCounts$_\varepsilon(X, Y)$ by $\widetilde{X}$, $\widetilde{Y}$ and take $\widetilde{Z}=\widetilde{X}/\wi

Figures (3)

  • Figure 1: Comparison of the sample accuracy $(1 - \beta)$ of the algorithms LaplaceNoisedCounts (see Claim \ref{['clm:acc_Alg_count_no_max']}), NoisedLog (see Claim \ref{['clm:acc_exp_naive']}), Propose-Test-Release (see Algorithm \ref{['alg:PTR']}), and the smooth sensitivity approach (described in Appendix \ref{['App:local_noise_methods']}). We take $\alpha=0.1$, and $\delta=1/n_x$ when needed.
  • Figure 2: Comparison of: the bias of a ratio of noisy counts drawn according to Equation \ref{['eq:bin_rv']} ($\widehat{p}$), a ratio of the outputs of LaplaceNoisedCounts$_\varepsilon(X, Y)$ on those counts ($\widetilde{p}$), and the expected value according to Claim \ref{['clm:bias_ratio_counts']}. For each $\varepsilon$ we sample $20,000$ pairs of binomial random variables with $n_x=n_y=150$ and $p_x$ and $p_y$ as stated in the title, and report the average for each $\varepsilon$.
  • Figure 3: Average width and coverage of 10,000 CIs constructed for 10,000 pairs of binomial random variables drawn according to Equation \ref{['eq:bin_rv']}, with $n_x=n_y=150$. We compute the Classic CI (see Equation \ref{['eq:conf_int_RR']}) and the non-private CI from binomial counts. We then perturb the counts with both Laplace and Gaussian noise to obtain $(\varepsilon,0)$-DP and $(\varepsilon, \delta=10^{-4})$-DP respectively, and compute CIs using Theorem \ref{['thm:Valid_CI']} (Private Laplace and Private Gaussian), and according to Theorem \ref{['thm:Gaussian_Valid_CI']} (Cons Private Gaussian and Laplace); for the Laplace noised counts we plug in the variance of the Laplace.

Theorems & Definitions (47)

  • Definition 2.1: Differential privacy
  • Definition 2.2: $(\alpha,\beta)$-sample accuracy
  • Definition 2.3: Approximate CI
  • Claim 3.1
  • Claim 3.2
  • Corollary 3.3
  • Claim 3.4: Bias of maxed ratio of noisy counts
  • Claim 4.1: Consistency of $\widetilde{p}$
  • proof
  • Theorem 4.2: Valid CI for the relative risk
  • ...and 37 more