SynQP: A Framework and Metrics for Evaluating the Quality and Privacy Risk of Synthetic Data

TL;DR

The paper addresses the lack of open privacy-evaluation frameworks for synthetic data in health by introducing SynQP, an open framework that benchmarks privacy in SDG using simulated pseudo-identifiable data. It defines two high-signal metrics, SD-IDR and SD-MIA, to account for the probabilistic nature of SDG models and attacker knowledge, and demonstrates them through a CTGAN case study with differential privacy. The methodology integrates quasi-identifier seeding, linked real-use-case data, local differential privacy, and standardized quality (fidelity and utility) and privacy evaluations. The work enables transparent, regulatory-aligned benchmarking of synthetic data, facilitating safer adoption of SDG in health research and policy contexts.

Abstract

The use of synthetic data in health applications raises privacy concerns, yet the lack of open frameworks for privacy evaluations has slowed its adoption. A major challenge is the absence of accessible benchmark datasets for evaluating privacy risks, due to difficulties in acquiring sensitive data. To address this, we introduce SynQP, an open framework for benchmarking privacy in synthetic data generation (SDG) using simulated sensitive data, ensuring that original data remains confidential. We also highlight the need for privacy metrics that fairly account for the probabilistic nature of machine learning models. As a demonstration, we use SynQP to benchmark CTGAN and propose a new identity disclosure risk metric that offers a more accurate estimation of privacy risks compared to existing approaches. Our work provides a critical tool for improving the transparency and reliability of privacy evaluations, enabling safer use of synthetic data in health-related applications. % In our quality evaluations, non-private models achieved near-perfect machine-learning efficacy \(\ge0.97\). Our privacy assessments (Table II) reveal that DP consistently lowers both identity disclosure risk (SD-IDR) and membership-inference attack risk (SD-MIA), with all DP-augmented models staying below the 0.09 regulatory threshold. Code available at https://github.com/CAN-SYNH/SynQP

Figures (2)

• Figure 1: SynQP workflow and framework for simulated data generation and synthetic data evaluation.
• Figure 2: Comparing training data corresponding generated synthetic data distributions for age and BMI with and without DP for CTGAN, TVAE, and GaussianCopula.