Computing Maximal Per-Record Leakage and Leakage-Distortion Functions for Privacy Mechanisms under Entropy-Constrained Adversaries

TL;DR

This work addresses privacy under entropy-constrained adversaries by formalizing an information privacy (IP) framework with $H(X) \ge b$, and defines three core optimization problems: maximal per-record leakage $\mathcal{L}(b)$, primal leakage-distortion $\epsilon(D,b)$, and dual minimal-distortion $\mathcal{D}(L,b)$. It develops efficient alternating-optimization algorithms that exploit the convex-concave duality of mutual information and enforces the entropy constraint, with local convergence for the leakage and primal problems and stationary-point convergence for the dual. The authors provide comprehensive convergence analyses and validate their methods on binary symmetric channels and modular-sum queries, showing improved privacy-utility tradeoffs over classical DP mechanisms. The framework enables precise auditing and design of certified mechanisms under bounded-knowledge adversaries, with practical implications for privacy-preserving data sharing and trustworthy AI.

Abstract

The exponential growth of data collection necessitates robust privacy protections that preserve data utility. We address information disclosure against adversaries with bounded prior knowledge, modeled by an entropy constraint $H(X) \geq b$. Within this information privacy framework -- which replaces differential privacy's independence assumption with a bounded-knowledge model -- we study three core problems: maximal per-record leakage, the primal leakage-distortion tradeoff (minimizing worst-case leakage under distortion $D$), and the dual distortion minimization (minimizing distortion under leakage constraint $L$). These problems resemble classical information-theoretic ones (channel capacity, rate-distortion) but are more complex due to high dimensionality and the entropy constraint. We develop efficient alternating optimization algorithms that exploit convexity-concavity duality, with theoretical guarantees including local convergence for the primal problem and convergence to a stationary point for the dual. Experiments on binary symmetric channels and modular sum queries validate the algorithms, showing improved privacy-utility tradeoffs over classical differential privacy mechanisms. This work provides a computational framework for auditing privacy risks and designing certified mechanisms under realistic adversary assumptions.

Figures (7)

• Figure 1: Saddle-shaped structure of mutual information under convex-concave duality. (Left) Basic saddle surface demonstrating the convex-concave duality of mutual information $I(X_i;Y)$: concave in the marginal distribution $p(x_i)$ and convex in the conditional distribution $p(x_{-i}|x_i)$. (Right) Saddle surface with entropy constraint $H(X) \geq b$ (red boundary), illustrating the feasible region for entropy-constrained optimization. Blue and green curves demonstrate cross-sections showing the concave/convex properties along fixed conditionals and marginals, respectively.
• Figure 2: Algorithm Dependency Graph: Core Methods for Maximal Leakage Computation, Leakage-Distortion Optimization, and Dual Formulation with Their Component Subroutines
• Figure 3: Variation of individual channel capacity $C_1^b$ with entropy constraint $b$, consisting of three subfigures corresponding to different numbers of records $n$ in the dataset: the left subfigure is for $n=4$, the middle subfigure for $n=5$, and the right subfigure for $n=6$. In each subfigure, there are four curves, which correspond to the generalized binary-symmetric privacy channel with different error probabilities $p$ as follows: the curve with the highest initial capacity $C_1^b$ corresponds to $p=0.1$, the next to $p=0.2$, then $p=0.3$, and the curve with the lowest initial capacity $C_1^b$ corresponds to $p=0.4$. The $x$-axis of each subfigure represents the entropy constraint $b$ (quantifying the lower bound of the adversary's knowledge uncertainty about the dataset, i.e., $H(X) \geq b$), and the $y$-axis denotes the individual channel capacity $C_1^b$ (measuring the maximum amount of information about any individual that an adversary can infer from the privacy channel output). All curves across the three subfigures consistently show that $C_1^b$ decreases as $b$ increases, confirming that the entropy constraint $H(X) \ge b$ effectively limits potential information leakage about any individual by constraining the adversary's prior knowledge. Additionally, for the same $n$ and $b$, a larger $p$ leads to a smaller $C_1^b$, as higher randomness injection further reduces the adversary's ability to infer individual information. The non-monotonic segments observed in all curves originate from the non-convex nature of the $C_1^b$ calculation optimization problem, where our algorithms converge to local optima temporarily during the alternating optimization process.
• Figure 4: Comparison of per-record capacity $C_1^b$ (maximum mutual information $I(X_i;Y)$ in nats) across three privacy mechanisms for different dataset sizes. From left to right: $n=4$, $n=5$, and $n=6$ records. In each subfigure, three mechanisms are shown: Laplace mechanism (blue curve), Exponential mechanism (red curve), and binary symmetric channel with crossover probability $p=0.3$ (BSC(0.3), shown as horizontal lines). The $x$-axis represents the privacy parameter $\epsilon$ (log scale) for the Laplace and Exponential mechanisms, while the $y$-axis denotes the per-record capacity. The Laplace and Exponential mechanisms exhibit the standard differential privacy trade-off where capacity decreases as $\epsilon$ increases (i.e., stronger privacy). For BSC(0.3), which does not depend on $\epsilon$, we show capacities under different entropy constraints $b = 0, 0.5, 1.0, 1.5, 2.0, 2.5$ nats (green to purple lines, with decreasing line thickness and varying styles). Each BSC(0.3) capacity value is computed by solving the optimization problem of maximizing $I(X_i;Y)$ subject to the entropy constraint $H(X) \geq b$ for the corresponding dataset size $n$. The results demonstrate that for a fixed $\epsilon$, the Laplace and Exponential mechanisms achieve higher capacity than BSC(0.3) with $b>0$, but when $b=0$ (no entropy constraint) BSC(0.3) has constant capacity $C_1^0 \approx 0.0931$ nats for $n=4$, decreasing for larger $n$. As $b$ increases, the capacity of BSC(0.3) decreases, reflecting the limitation imposed by the adversary's uncertainty about the dataset. The horizontal dotted line at $\ln(2) \approx 0.693$ nats represents the theoretical maximum capacity for any binary channel. The non-monotonic segments in some curves originate from the non-convex nature of the $C_1^b$ optimization problem, where the alternating algorithm may converge to local optima despite multiple random restarts.
• Figure 5: Comparison of individual channel capacity $C_1^b$ versus expected distortion $\mathbb E_{p_0} d(f(X),Y)$ across three privacy mechanisms for the binary parity query function, consisting of three subfigures corresponding to different numbers of records $n$ in the dataset: the left subfigure is for $n=4$ (with maximum entropy of 2.77 nats), the middle subfigure for $n=5$ (with maximum entropy of 3.47 nats), and the right subfigure for $n=6$ (with maximum entropy of 4.159 nats). In each subfigure, there are three kinds of curves, which correspond to different privacy mechanisms as follows: the four dotted curves with the lower privacy leakages at the same distortion level corresponds to the binary symmetric privacy channel, the next curve corresponds to the Laplace mechanism, and the curve with the highest privacy leakage corresponds to the Exponential mechanism. The $x$-axis denotes the maximal privacy leakage/privacy budget, measured by the privacy parameter $\epsilon$: for the binary symmetric privacy channel, $\epsilon=C_1^b$ to ensure it satisfies $\epsilon$-Information Privacy with respect to the adversary class $\mathbb{P}_b$; for fair comparison, the privacy budget of the Laplace mechanism and Exponential mechanism is set equal to $\epsilon$ to guarantee $\epsilon$-Differential Privacy. The $y$-axis of each subfigure represents the expected distortion $\mathbb E_{p_0} d(f(X),Y)$. All curves across the three subfigures consistently show two key results: (1) Under the same expected distortion, the binary symmetric privacy channel requires a smaller privacy budget $\epsilon$ (i.e., lower information leakage) than both the Laplace mechanism and the Exponential mechanism; (2) Increasing the entropy constraint $b$ further reduces the privacy leakage $C_1^b$ of the binary symmetric privacy channel at the same distortion level, which highlights the utility advantage of the information privacy framework enabled by the bounded knowledge assumption ($H(X) \geq b$).

Theorems & Definitions (24)

• Definition 2.1: Maximal Per-Record Leakage
• Definition 2.2: Primal Leakage-Distortion Tradeoff
• Definition 2.3: Dual Minimal-Distortion Formulation
• Theorem 1: Mutual Information Convexity-Concavity Duality
• proof
• Theorem 2: Convexity of Constraint Sets
• proof
• Lemma 1: Representation of Entropy Super-Level Sets
• proof
• Theorem 3: Optimality on Entropy Boundary