What's Wrong with Your Synthetic Tabular Data? Using Explainable AI to Evaluate Generative Models

TL;DR

This work tackles the problem of evaluating synthetic tabular data by introducing explainable AI (XAI) as a diagnostic aid for synthetic-data quality. It trains a binary detector C: $\,\mathcal{X}\to[0,1]$ to distinguish real from synthetic data and then applies a suite of XAI techniques—permutation feature importance, partial dependence/ICE plots, conditional and marginal Shapley values, and counterfactuals—to identify which features, dependencies, and regions of the data space drive detectability. The approach reveals concrete fidelity and diversity weaknesses in synthetic data produced by TabSyn and CTGAN on two real-world datasets, uncovering issues that standard metrics overlook and offering actionable insights to improve generative models. The study emphasizes the value of explanatory diagnostics for auditing synthetic data pipelines and guiding model improvements, while acknowledging limitations related to detector quality and the maturity of tabular-generation methods.

Abstract

Evaluating synthetic tabular data is challenging, since they can differ from the real data in so many ways. There exist numerous metrics of synthetic data quality, ranging from statistical distances to predictive performance, often providing conflicting results. Moreover, they fail to explain or pinpoint the specific weaknesses in the synthetic data. To address this, we apply explainable AI (XAI) techniques to a binary detection classifier trained to distinguish real from synthetic data. While the classifier identifies distributional differences, XAI concepts such as feature importance and feature effects, analyzed through methods like permutation feature importance, partial dependence plots, Shapley values and counterfactual explanations, reveal why synthetic data are distinguishable, highlighting inconsistencies, unrealistic dependencies, or missing patterns. This interpretability increases transparency in synthetic data evaluation and provides deeper insights beyond conventional metrics, helping diagnose and improve synthetic data quality. We apply our approach to two tabular datasets and generative models, showing that it uncovers issues overlooked by standard evaluation techniques.

Figures (11)

• Figure 1: (a) Synthetic data detection performance of logistic regression, random forest breiman2001RF and XGBoost models with six generative models generating five synthetic datasets for eleven original datasets each. CTAB-GAN+ did not converge for all runs. (b) Synthetic data detection performance for XGBoost on train and test data for adult and nursery data with ten replications.
• Figure 2: Feature importance values for synthetic data detection with XGBoost for ten TabSyn-generated synthetic adult datasets. Higher importance values indicate poorer synthesis quality. (a) PFI and global TreeSHAP values. (b) Global TreeSHAP interaction values of degree 1 and 2 (top 20 most important).
• Figure 3: ICE/PDP for synthetic data detection with XGBoost for TabSyn-generated synthetic adult data. (a) Numeric feature education_num, distribution for original and synthetic data on x-axis. (b) Categorical feature occupation, PDP in red, frequencies for real and synthetic data on y-axis.
• Figure 4: Force plots for conditional and marginal Shapley values decomposing the XGBoost prediction for an exemplary instance of Tabsyn-generated synthetic adult data.
• Figure 5: Waterfall plots for Shapley interaction values decomposing the XGBoost prediction for an examplary instance of TabSyn-generated synthetic adult data. Note that TreeSHAP decomposes the prediction on the log-odds/logistic scale rather than on the probability scale.