When Privacy Isn't Synthetic: Hidden Data Leakage in Generative AI Models

TL;DR

This paper reveals that privacy-preserving synthetic data can still leak information about the training data through distributional overlap, even under black-box access. It introduces the Cluster–Medoid Leakage Attack (CMLA), a model-agnostic framework that samples synthetic outputs, encodes them into a shared space, clusters them, and extracts medoids to measure proximity to real data via $d_{ ext{min}}$, ASR(τ), and Cov(τ). Empirical results across healthcare, finance, and other sensitive domains show leakage patterns across a range of generative models, including those with differential privacy, challenging the assumption that synthetic data fully protects privacy. The work provides practical privacy-auditing tools and argues for stronger guarantees that address neighborhood-level leakage, not just memorization, with publicly available code to foster adoption and evaluation.

Abstract

Generative models are increasingly used to produce privacy-preserving synthetic data as a safe alternative to sharing sensitive training datasets. However, we demonstrate that such synthetic releases can still leak information about the underlying training samples through structural overlap in the data manifold. We propose a black-box membership inference attack that exploits this vulnerability without requiring access to model internals or real data. The attacker repeatedly queries the generative model to obtain large numbers of synthetic samples, performs unsupervised clustering to identify dense regions of the synthetic distribution, and then analyzes cluster medoids and neighborhoods that correspond to high-density regions in the original training data. These neighborhoods act as proxies for training samples, enabling the adversary to infer membership or reconstruct approximate records. Our experiments across healthcare, finance, and other sensitive domains show that cluster overlap between real and synthetic data leads to measurable membership leakage-even when the generator is trained with differential privacy or other noise mechanisms. The results highlight an under-explored attack surface in synthetic data generation pipelines and call for stronger privacy guarantees that account for distributional neighborhood inference rather than sample-level memorization alone, underscoring its role in privacy-preserving data publishing. Implementation and evaluation code are publicly available at:github.com/Cluster-Medoid-Leakage-Attack.

Figures (4)

• Figure 1: Visualization of how synthetic data can unintentionally reproduce the structure of its training data. Even though the generative model produces “new” records, their positions reveal that the generator has learned specific structures from the original dataset. These structural echoes enable an attacker to infer information about the training data without ever accessing it directly.
• Figure 2: Conceptual overview of the black-box cluster-overlap attack. The attacker repeatedly queries the synthetic data generator to collect synthetic samples, clusters them to uncover dense regions, and identifies cluster centers (medoids). Overlaps between synthetic and real data clusters indicate memorization or membership leakage, even when the attacker never sees the real training data.
• Figure 3: Attack success vs. threshold. The $y$-axis shows ASR at threshold $\tau$: the fraction of cluster medoids whose nearest real neighbor is within distance $\tau$ (i.e., $d_{\min}<\tau$). Each line is a generator. The $x$-axis sweeps increasingly permissive thresholds; higher curves (especially at small $\tau$) indicate more medoids that are unusually close to real records and thus greater leakage risk. The dashed reference marks $\tau=0.1$.
• Figure 4: Overall coverage Each cell reports the estimated share of the REAL dataset that lies within distance $\tau$ of at least one synthetic medoid from a generator (higher = riskier). Read across columns to compare datasets and down rows to compare generators. Darker cells mean synthetic exemplars sit close to a larger portion of the real population—already at a tight radius (left, $\tau=0.1$) and more so at a looser radius (right, $\tau=0.5$).