Synthetic Data -- Anonymisation Groundhog Day

TL;DR

The paper rigorously evaluates the privacy benefits of publishing synthetic data against traditional anonymisation for high-dimensional tabular data. It introduces a model-agnostic evaluation framework that quantifies privacy gain via linkage and attribute-inference adversaries, and applies it to five generative models, including privacy-preserving variants. The empirical results show that synthetic data does not reliably outperform sanitisation: outliers remain vulnerable to linkage attacks, and differential privacy requirements introduce substantial utility losses with unpredictable tradeoffs. The authors also reveal implementation pitfalls in DP methods and provide an open-source toolkit for practitioners to assess privacy-utility tradeoffs, emphasizing that synthetic data is not a universal privacy solution. Overall, the work cautions against assuming synthetic data automatically achieves better privacy-utility outcomes and highlights the need for careful design, evaluation, and transparency.

Abstract

Synthetic data has been advertised as a silver-bullet solution to privacy-preserving data publishing that addresses the shortcomings of traditional anonymisation techniques. The promise is that synthetic data drawn from generative models preserves the statistical properties of the original dataset but, at the same time, provides perfect protection against privacy attacks. In this work, we present the first quantitative evaluation of the privacy gain of synthetic data publishing and compare it to that of previous anonymisation techniques. Our evaluation of a wide range of state-of-the-art generative models demonstrates that synthetic data either does not prevent inference attacks or does not retain data utility. In other words, we empirically show that synthetic data does not provide a better tradeoff between privacy and utility than traditional anonymisation techniques. Furthermore, in contrast to traditional anonymisation, the privacy-utility tradeoff of synthetic data publishing is hard to predict. Because it is impossible to predict what signals a synthetic dataset will preserve and what information will be lost, synthetic data leads to a highly variable privacy gain and unpredictable utility loss. In summary, we find that synthetic data is far from the holy grail of privacy-preserving data publishing.

Figures (15)

• Figure 1: Linkability privacy game.
• Figure 2: Expected per-record privacy gain for outliers and random records for the Texas (top row) and Adult (bottom row) datasets under three different attacks using three distinct feature sets. Error bars represent the standard deviation.
• Figure 3: Per-record privacy gain for five outlier targets records from the Texas (top row) and Adult (bottom row) datasets under an attack using the $\mathtt{F_{Hist}}$ feature set.
• Figure 4: Per-record privacy gain for five outlier target records from the Texas dataset under three different attacks using three distinct feature sets.
• Figure 5: Attribute inference privacy game