Synthetic Tabular Data Validation: A Divergence-Based Approach

TL;DR

The paper introduces a divergence-based framework for validating synthetic tabular data by estimating the density ratio between real and synthetic distributions using a probabilistic classifier. It computes the Kullback-Leibler and Jensen-Shannon divergences to capture joint-distribution differences, addressing limitations of marginal metrics. Through controlled experiments and real-world data with CTGAN and VAE baselines, the approach shows that sufficient training data yields accurate divergence estimates and that JS is particularly robust due to its bounded nature. The methodology offers a path toward standardized, interpretable validation of synthetic data with potential applicability beyond tabular domains.

Abstract

The ever-increasing use of generative models in various fields where tabular data is used highlights the need for robust and standardized validation metrics to assess the similarity between real and synthetic data. Current methods lack a unified framework and rely on diverse and often inconclusive statistical measures. Divergences, which quantify discrepancies between data distributions, offer a promising avenue for validation. However, traditional approaches calculate divergences independently for each feature due to the complexity of joint distribution modeling. This paper addresses this challenge by proposing a novel approach that uses divergence estimation to overcome the limitations of marginal comparisons. Our core contribution lies in applying a divergence estimator to build a validation metric considering the joint distribution of real and synthetic data. We leverage a probabilistic classifier to approximate the density ratio between datasets, allowing the capture of complex relationships. We specifically calculate two divergences: the well-known Kullback-Leibler (KL) divergence and the Jensen-Shannon (JS) divergence. KL divergence offers an established use in the field, while JS divergence is symmetric and bounded, providing a reliable metric. The efficacy of this approach is demonstrated through a series of experiments with varying distribution complexities. The initial phase involves comparing estimated divergences with analytical solutions for simple distributions, setting a benchmark for accuracy. Finally, we validate our method on a real-world dataset and its corresponding synthetic counterpart, showcasing its effectiveness in practical applications. This research offers a significant contribution with applicability beyond tabular data and the potential to improve synthetic data validation in various fields.

Figures (10)

• Figure 1: Architecture of the neural network-based divergence estimator to assess the dissimilarity between samples from two datasets. The discriminator takes two sets of samples as input: $M$ samples from each set to train and $L$ samples from each to infer the divergence.
• Figure 2: Generative model for divergence estimation between real and synthetic data. The generative model learns an approximation of $N$ samples from real data $x_r$, denoted $x_s$. Subsequently, $M$ samples are drawn from each distribution to train the divergence estimation discriminator. Finally, $L$ samples from each distribution estimate the divergence between real and synthetic data.
• Figure 3: Estimation error representation for $D_\mathbb{KL}$ in Experiment 1. Results are shown for different combinations of training sample sizes $M$ and validation sample sizes $L$. As expected, a decrease and precision in the error is observed with increasing values of $M$ and $L$.
• Figure 4: Estimation error representation for $D_\mathbb{JS}$ in Experiment 1. Results are shown for different combinations of training sample sizes $M$ and validation sample sizes $L$. As expected, a decrease and precision in the error is observed with increasing values of $M$ and $L$.
• Figure 5: Discriminator loss curves for Experiment 1. The loss curves show a clear overfitting due to low sample sizes. Green and red dashed lines represent theoretical convergence values.