Metrics for Inter-Dataset Similarity with Example Applications in Synthetic Data and Feature Selection Evaluation -- Extended Version

TL;DR

The paper tackles measuring inter-dataset similarity by introducing two PCA-based metrics, Δλ (differences in explained variance) and Δθ (differences in the direction of the first principal component), and grounding them in PCA theory. It provides formal definitions, discusses invariances, and analyzes stability with respect to sample size. The authors validate the metrics through two applications—synthetic data evaluation and feature-selection evaluation—highlighting their model-agnostic nature and potential for privacy sanity checks. The work suggests these metrics capture global multivariate similarity and can inform dataset selection, synthetic data quality, and feature-selection practices across ML tasks, while outlining limitations and avenues for future extension to non-numeric data and broader domains.

Abstract

Measuring inter-dataset similarity is an important task in machine learning and data mining with various use cases and applications. Existing methods for measuring inter-dataset similarity are computationally expensive, limited, or sensitive to different entities and non-trivial choices for parameters. They also lack a holistic perspective on the entire dataset. In this paper, we propose two novel metrics for measuring inter-dataset similarity. We discuss the mathematical foundation and the theoretical basis of our proposed metrics. We demonstrate the effectiveness of the proposed metrics by investigating two applications in the evaluation of synthetic data and in the evaluation of feature selection methods. The theoretical and empirical studies conducted in this paper illustrate the effectiveness of the proposed metrics.

Figures (7)

• Figure 1: Samples from Increasingly Similar Populations. The figure shows a source distribution (in blue) and a query distribution (in red). From left to right, the query distribution is moved closer to the source distribution, changing the values of the proposed metrics $\Delta\lambda$ and $\Delta\theta$.
• Figure 2: Metric Variation with Varying Sample Size and Number of Attributes. The proposed metrics and other state-of-the-art metrics (correlation matrix difference, propensity mean squared error, and the Kolmogorov-Smirnov test) do not conform to the optimum value, due to noise for fewer samples and sparsity from additional dimensions even when the compared samples stem from the same multivariate distribution.
• Figure 3: TVAE Training Loss Alongside the Proposed Metrics. After an initial period (the model learns characteristics of the data), the metrics drop, indicating a new alignment of the real and synthetic data. Afterwards the metrics steadily improve, aligned with the decreased loss. This demonstrates that the metrics are meaningful for measuring synthetic data quality. Confidence interval is 95%, based on 10 independent runs.
• Figure 4: Correlation Hierarchy of Metrics. The heat-map is produced by taking the correlations of results of the metrics on 64 synthetic datasets made using different generative models. Closely associated metrics have little value in the same benchmarks since they describe the same mode of similarity/privacy leakage. The values of each metric were normalized such that a high value is better. The proposed metrics are outlined in blue.
• Figure 5: Experimental Results for Feature selection. The experimental results on 20 different datasets, 4 different feature selection methods and compared to two different model-dependent metric.