Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differential Privacy via Distributionally Robust Optimization

Aras Selvi, Huikang Liu, Wolfram Wiesemann

TL;DR

This work reframes differential privacy mechanism design as an infinite-dimensional distributionally robust optimization problem to achieve non-asymptotic, instance-specific optimality. By leveraging strong duality, it builds convergent hierarchies of finite-dimensional upper and lower bounds for both data-independent and data-dependent noise designs, solvable via tailored cutting plane methods. The approach yields implementable noise perturbations that can outperform classical mechanisms like Laplace or Gaussian in various privacy regimes and extends to ML tasks such as private Naïve Bayes and proximal coordinate descent. The framework offers a flexible, extensible toolkit for DP mechanism design, with potential extensions to multi-dimensional queries and broader DP notions, underscoring the intersection of optimization and privacy.

Abstract

In recent years, differential privacy has emerged as the de facto standard for sharing statistics of datasets while limiting the disclosure of private information about the involved individuals. This is achieved by randomly perturbing the statistics to be published, which in turn leads to a privacy-accuracy trade-off: larger perturbations provide stronger privacy guarantees, but they result in less accurate statistics that offer lower utility to the recipients. Of particular interest are therefore optimal mechanisms that provide the highest accuracy for a pre-selected level of privacy. To date, work in this area has focused on specifying families of perturbations a priori and subsequently proving their asymptotic and/or best-in-class optimality. In this paper, we develop a class of mechanisms that enjoy non-asymptotic and unconditional optimality guarantees. To this end, we formulate the mechanism design problem as an infinite-dimensional distributionally robust optimization problem. We show that the problem affords a strong dual, and we exploit this duality to develop converging hierarchies of finite-dimensional upper and lower bounding problems. Our upper (primal) bounds correspond to implementable perturbations whose suboptimality can be bounded by our lower (dual) bounds. Both bounding problems can be solved within seconds via cutting plane techniques that exploit the inherent problem structure. Our numerical experiments demonstrate that our perturbations can outperform the previously best results from the literature on artificial as well as standard benchmark problems.

Differential Privacy via Distributionally Robust Optimization

TL;DR

This work reframes differential privacy mechanism design as an infinite-dimensional distributionally robust optimization problem to achieve non-asymptotic, instance-specific optimality. By leveraging strong duality, it builds convergent hierarchies of finite-dimensional upper and lower bounds for both data-independent and data-dependent noise designs, solvable via tailored cutting plane methods. The approach yields implementable noise perturbations that can outperform classical mechanisms like Laplace or Gaussian in various privacy regimes and extends to ML tasks such as private Naïve Bayes and proximal coordinate descent. The framework offers a flexible, extensible toolkit for DP mechanism design, with potential extensions to multi-dimensional queries and broader DP notions, underscoring the intersection of optimization and privacy.

Abstract

In recent years, differential privacy has emerged as the de facto standard for sharing statistics of datasets while limiting the disclosure of private information about the involved individuals. This is achieved by randomly perturbing the statistics to be published, which in turn leads to a privacy-accuracy trade-off: larger perturbations provide stronger privacy guarantees, but they result in less accurate statistics that offer lower utility to the recipients. Of particular interest are therefore optimal mechanisms that provide the highest accuracy for a pre-selected level of privacy. To date, work in this area has focused on specifying families of perturbations a priori and subsequently proving their asymptotic and/or best-in-class optimality. In this paper, we develop a class of mechanisms that enjoy non-asymptotic and unconditional optimality guarantees. To this end, we formulate the mechanism design problem as an infinite-dimensional distributionally robust optimization problem. We show that the problem affords a strong dual, and we exploit this duality to develop converging hierarchies of finite-dimensional upper and lower bounding problems. Our upper (primal) bounds correspond to implementable perturbations whose suboptimality can be bounded by our lower (dual) bounds. Both bounding problems can be solved within seconds via cutting plane techniques that exploit the inherent problem structure. Our numerical experiments demonstrate that our perturbations can outperform the previously best results from the literature on artificial as well as standard benchmark problems.
Paper Structure (40 sections, 30 theorems, 135 equations, 8 figures, 6 tables, 3 algorithms)

This paper contains 40 sections, 30 theorems, 135 equations, 8 figures, 6 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Data Independent Noise Optimization
  3. The Data Independent Optimization Problem
  4. A Hierarchy of Converging Bounding Problems
  5. Data Dependent Noise Optimization
  6. The Data Dependent Optimization Problem
  7. A Hierarchy of Converging Bounding Problems
  8. Iterative Solution of the Bounding Problems
  9. Upper and Lower Bounding Problems with Non-Uniform Partitions
  10. Cutting Plane Algorithm
  11. Numerical Experiments
  12. Synthetic Experiments
  13. Differentially Private Naïve Bayes Classification
  14. Differentially Private Proximal Coordinate Descent
  15. Conclusions
  16. ...and 25 more sections

Key Result

lemma 1

Under restriction restriction, problem:integral_main has the same optimal value as where $c_i(\beta):= \beta^{-1} \cdot \int_{x \in I_i(\beta)} c(x) \mathrm{d}x$ and $\mathcal{E}(\beta) := \setvarphi(\beta) \times \mathcal{F}(\beta)$ with $\setvarphi(\beta) := \{-\Delta f, -\Delta f+ \beta, \ldots, \Delta f \}$ and $\mathcal{F}(\beta) := \{ \bigcup_{i \in \mathcal{I}}I_i(\beta) \

Figures (8)

  • Figure 1: Different noise distributions that guarantee $(1,0.2)$-DP in Example \ref{['example:recurred']}.
  • Figure 2: Summary of the results in Section \ref{['sec:additive']}. Directed arrows $x \dashrightarrow y$ indicate upper bound relationships $x \leq y$, whereas the double arrow confirms the convergence of optimal values as $L$ increases and $\beta$ decreases.
  • Figure 3: Comparison of our optimization-based DP schemes with benchmark approaches from the literature on instances with $\Delta f = 2$, $\ell_1$-loss and $| \Phi | = 4$ in low-privacy ($\varepsilon = 5$ and $\delta = 0.25$; left), medium-privacy ($\varepsilon = 1$ and $\delta = 0.2$; middle) and high-privacy ($\varepsilon = 0.2$ and $\delta = 0.05$; right) regimes. All computation times are median values over $10$ repetitions.
  • Figure 4: Optimization-based noise distributions for synthetic data independent instances with $\Delta f = 10$, $\beta = 0.5$, $\ell_1$-loss and various combinations of $\varepsilon$ and $\delta$. The unconstrained distributions are shown in red shading, whereas the best monotone distributions are shown as blue lines.
  • Figure 5: Optimization-based noise distributions for synthetic data dependent instances with $\Delta f = 10$, $\beta = 0.5$, $\ell_1$-loss and various combinations of $\varepsilon$ and $\delta$. The set $\Phi = [0, 18)$ of query outputs has been partitioned into $9$ intervals of equal length, resulting in $9$ noise distributions.
  • ...and 3 more figures

Theorems & Definitions (45)

  • lemma 1
  • proposition 1
  • proposition 2: Weak Duality
  • lemma 2
  • proposition 3
  • theorem 1
  • proposition 4
  • proposition 5
  • proposition 6: Weak Duality
  • proposition 7
  • ...and 35 more