Synthetic Dataset Evaluation Based on Generalized Cross Validation

TL;DR

The paper tackles the absence of a standard framework for evaluating synthetic datasets by introducing a generalized cross-validation approach that leverages domain transfer learning. It constructs a Cross-Performance Matrix and a normalized Generalized Cross-Validation Matrix to quantify both realism ($A_o$) and transfer quality ($S_o$) across real-world benchmarks. Through experiments on Virtual KITTI, KITTI, and BDD100K, it demonstrates measurable, interpretable metrics that capture fidelity and generalizability of synthetic data for object detection. The framework offers a principled, scalable path to optimize synthetic data generation and selection for AI research and deployment.

Abstract

With the rapid advancement of synthetic dataset generation techniques, evaluating the quality of synthetic data has become a critical research focus. Robust evaluation not only drives innovations in data generation methods but also guides researchers in optimizing the utilization of these synthetic resources. However, current evaluation studies for synthetic datasets remain limited, lacking a universally accepted standard framework. To address this, this paper proposes a novel evaluation framework integrating generalized cross-validation experiments and domain transfer learning principles, enabling generalizable and comparable assessments of synthetic dataset quality. The framework involves training task-specific models (e.g., YOLOv5s) on both synthetic datasets and multiple real-world benchmarks (e.g., KITTI, BDD100K), forming a cross-performance matrix. Following normalization, a Generalized Cross-Validation (GCV) Matrix is constructed to quantify domain transferability. The framework introduces two key metrics. One measures the simulation quality by quantifying the similarity between synthetic data and real-world datasets, while another evaluates the transfer quality by assessing the diversity and coverage of synthetic data across various real-world scenarios. Experimental validation on Virtual KITTI demonstrates the effectiveness of our proposed framework and metrics in assessing synthetic data fidelity. This scalable and quantifiable evaluation solution overcomes traditional limitations, providing a principled approach to guide synthetic dataset optimization in artificial intelligence research.

Figures (2)

• Figure 1: The structure of the generalized cross-validation evaluation framework.
• Figure 2: Example of evaluating synthetic dataset and reference real datasets ( left to right: BDD100Kbdd100k, KITTIkitti, and Virtual KITTIvkitti).