Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Cross Entropy Interpretation of R{é}nyi Entropy for $α$-leakage

Ni Ding, Mohammad Amin Zarrabian, Parastoo Sadeghi

TL;DR

The paper addresses information leakage across the whole Rényi order by presenting a cross-entropy interpretation of Rényi entropy. It shows that Rényi entropy is a $\tilde{f}$-mean cross entropy, and that minimizing this cross entropy yields the optimal estimator characterized by $P_{X_α}$. Using this, the authors define $α$-leakage as the reduction in $\tilde{f}$-mean uncertainty, which they prove equals the Arimoto mutual information $I_α^{A}(U;Y)$ and extends to $α\in[0,\infty)$ with $α=0$ capturing nonstochastic leakage. They further relate Sibson mutual information to a $\tilde{f}$-mean of elementary leakages and show maximal leakage is a $\tilde{f}$-mean of pointwise maximal leakage, unifying several leakage notions and offering a solid theoretical foundation for privacy metrics across all Rényi orders.

Abstract

This paper proposes an $α$-leakage measure for $α\in[0,\infty)$ by a cross entropy interpretation of R{é}nyi entropy. While Rényi entropy was originally defined as an $f$-mean for $f(t) = \exp((1-α)t)$, we reveal that it is also a $\tilde{f}$-mean cross entropy measure for $\tilde{f}(t) = \exp(\frac{1-α}αt)$. Minimizing this Rényi cross-entropy gives Rényi entropy, by which the prior and posterior uncertainty measures are defined corresponding to the adversary's knowledge gain on sensitive attribute before and after data release, respectively. The $α$-leakage is proposed as the difference between $\tilde{f}$-mean prior and posterior uncertainty measures, which is exactly the Arimoto mutual information. This not only extends the existing $α$-leakage from $α\in [1,\infty)$ to the overall R{é}nyi order range $α\in [0,\infty)$ in a well-founded way with $α=0$ referring to nonstochastic leakage, but also reveals that the existing maximal leakage is a $\tilde{f}$-mean of an elementary $α$-leakage for all $α\in [0,\infty)$, which generalizes the existing pointwise maximal leakage.

A Cross Entropy Interpretation of R{é}nyi Entropy for $α$-leakage

TL;DR

The paper addresses information leakage across the whole Rényi order by presenting a cross-entropy interpretation of Rényi entropy. It shows that Rényi entropy is a -mean cross entropy, and that minimizing this cross entropy yields the optimal estimator characterized by . Using this, the authors define -leakage as the reduction in -mean uncertainty, which they prove equals the Arimoto mutual information and extends to with capturing nonstochastic leakage. They further relate Sibson mutual information to a -mean of elementary leakages and show maximal leakage is a -mean of pointwise maximal leakage, unifying several leakage notions and offering a solid theoretical foundation for privacy metrics across all Rényi orders.

Abstract

This paper proposes an -leakage measure for by a cross entropy interpretation of R{é}nyi entropy. While Rényi entropy was originally defined as an -mean for , we reveal that it is also a -mean cross entropy measure for . Minimizing this Rényi cross-entropy gives Rényi entropy, by which the prior and posterior uncertainty measures are defined corresponding to the adversary's knowledge gain on sensitive attribute before and after data release, respectively. The -leakage is proposed as the difference between -mean prior and posterior uncertainty measures, which is exactly the Arimoto mutual information. This not only extends the existing -leakage from to the overall R{é}nyi order range in a well-founded way with referring to nonstochastic leakage, but also reveals that the existing maximal leakage is a -mean of an elementary -leakage for all , which generalizes the existing pointwise maximal leakage.
Paper Structure (17 sections, 3 theorems, 34 equations, 1 figure)

This paper contains 17 sections, 3 theorems, 34 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Notation
  3. Rényi Entropy: An $\tilde{f}$-mean Measure
  4. $\alpha$-Cross Entropy
  5. Difference between $\alpha \in (0,1)$ and $\alpha \in (1,\infty)$
  6. Optimal decision $\mathbf{P}_{X_\alpha}$ in $\alpha$
  7. Optimal Estimation $\mathbf{P}_{\hat{X}|Y}^*$
  8. Arimoto Conditional Entropy
  9. Comparison to $\alpha$-loss in Liao2019_AlphaLeak
  10. Other cross entropy measures
  11. $\alpha$-leakage
  12. Maximal $\alpha$-leakage
  13. $Y$-elementary $\alpha$-leakage
  14. Comparison to $\alpha$-leakage in Liao2019_AlphaLeak
  15. Conclusion
  16. ...and 2 more sections

Key Result

Theorem 1

For a given $\mathbf{P}_X$, with the minimizer $\mathbf{P}_{\hat{X}}^* = \mathbf{P}_{X_\alpha}$ for all $\alpha \in [0,\infty)$.

Figures (1)

  • Figure 1: For $X \sim \text{Binomial}(20,5)$, the probability $\mathbf{P}_{X_\alpha}$ with $P_{X_\alpha}(x) = \frac{P_X^\alpha(x)}{\sum_{x\in\mathcal{X}}P_X^\alpha(x)}$ for all $x \in \{0,1,\dotsc,20\}$. Here, the plot for $\alpha = 1$ shows the original Binomial probability.

Theorems & Definitions (4)

  • Theorem 1
  • Definition 1
  • Proposition 1
  • Corollary 1