The DCR Delusion: Measuring the Privacy Risk of Synthetic Data

TL;DR

The paper addresses the challenge of quantifying privacy risk in synthetic data and critiques the widespread use of distance-based proxy metrics like Distance to Closest Record (DCR). It systematically compares proxy metrics with state-of-the-art Membership Inference Attacks (MIAs) across classical and diffusion tabular data generators, showing that proxies often misclassify leaked datasets as private and exhibit little correlation with true risk. The study demonstrates leakage in both classical models (e.g., CTGAN, BayNet) and diffusion models (e.g., TabDDPM, ClavaDDPM), including cases where uniquely identifying features enable membership disclosure that proxy tests fail to detect. The authors argue for abandoning proxy-based privacy claims in favor of rigorous MIA-based evaluation to ensure robust privacy guarantees, particularly for legal anonymity claims in synthetic data releases.

Abstract

Synthetic data has become an increasingly popular way to share data without revealing sensitive information. Though Membership Inference Attacks (MIAs) are widely considered the gold standard for empirically assessing the privacy of a synthetic dataset, practitioners and researchers often rely on simpler proxy metrics such as Distance to Closest Record (DCR). These metrics estimate privacy by measuring the similarity between the training data and generated synthetic data. This similarity is also compared against that between the training data and a disjoint holdout set of real records to construct a binary privacy test. If the synthetic data is not more similar to the training data than the holdout set is, it passes the test and is considered private. In this work we show that, while computationally inexpensive, DCR and other distance-based metrics fail to identify privacy leakage. Across multiple datasets and both classical models such as Baynet and CTGAN and more recent diffusion models, we show that datasets deemed private by proxy metrics are highly vulnerable to MIAs. We similarly find both the binary privacy test and the continuous measure based on these metrics to be uninformative of actual membership inference risk. We further show that these failures are consistent across different metric hyperparameter settings and record selection methods. Finally, we argue DCR and other distance-based metrics to be flawed by design and show a example of a simple leakage they miss in practice. With this work, we hope to motivate practitioners to move away from proxy metrics to MIAs as the rigorous, comprehensive standard of evaluating privacy of synthetic data, in particular to make claims of datasets being legally anonymous.

Figures (6)

• Figure 1: Extended-TAPAS MIA AUC on datasets considered "private" by $\tau_{\text{DCR,NNDR,IMS}}$ across classical SDG setups. Each dataset-SDG setup contains 100 target records selected using the Achilles score.
• Figure 2: Comparison of mean $\mu_{\text{DCR}}$ and MIA AUC for the Baynet generator on the Adult dataset. Each point represents a target record's MIA AUC and its mean $\mu_{\text{DCR}}$ across evaluation datasets.
• Figure 3: Distribution of TPR@FPR=0% for MIDST attacks against TabDDPM and ClavaDDPM.
• Figure 4: Comparison of $d_{\text{DCR}}(D_{\text{synthetic}},D_{\text{target}})-d_{\text{DCR}}(D_{\text{holdout}},D_{\text{target}})$ across percentile thresholds for a synthetic dataset trained on a vulnerable record with MIA AUC = 0.84.
• Figure 5: MIA AUCs of all 1000 target records and 100 vulnerable records selected using Achilles in the Adult-Baynet setup, in synthetic datasets considered "private" by $\tau_{\text{DCR,NNDR,IMS}}$.