Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Generative AI for Sim2Real in Driving Data Synthesis

Haonan Zhao, Yiting Wang, Thomas Bashford-Rogers, Valentina Donzella, Kurt Debattista

TL;DR

The paper tackles the Sim2Real challenge in driving data synthesis by evaluating three generative AI models on semantic label maps derived from driving simulators. It systematically compares two GAN-based I2I methods (Pix2pixHD, OASIS) and a diffusion-based approach (ControlNet) using both Cityscapes and SHIFT-derived labels, across image-quality and perception metrics. The results show GANs excel for Cityscapes-aligned realism, while ControlNet offers fewer artefacts and better structural fidelity when using simulator-generated labels, with diffusion-based methods showing promising stability and generalization. This work demonstrates that diffusion-guided, text- and layout-conditioned generation can be a practical, scalable path to realistic, richly annotated driving datasets, potentially reducing reliance on costly real-world data.

Abstract

Datasets are essential for training and testing vehicle perception algorithms. However, the collection and annotation of real-world images is time-consuming and expensive. Driving simulators offer a solution by automatically generating various driving scenarios with corresponding annotations, but the simulation-to-reality (Sim2Real) domain gap remains a challenge. While most of the Generative Artificial Intelligence (AI) follows the de facto Generative Adversarial Nets (GANs)-based methods, the recent emerging diffusion probabilistic models have not been fully explored in mitigating Sim2Real challenges for driving data synthesis. To explore the performance, this paper applied three different generative AI methods to leverage semantic label maps from a driving simulator as a bridge for the creation of realistic datasets. A comparative analysis of these methods is presented from the perspective of image quality and perception. New synthetic datasets, which include driving images and auto-generated high-quality annotations, are produced with low costs and high scene variability. The experimental results show that although GAN-based methods are adept at generating high-quality images when provided with manually annotated labels, ControlNet produces synthetic datasets with fewer artefacts and more structural fidelity when using simulator-generated labels. This suggests that the diffusion-based approach may provide improved stability and an alternative method for addressing Sim2Real challenges.

Exploring Generative AI for Sim2Real in Driving Data Synthesis

TL;DR

The paper tackles the Sim2Real challenge in driving data synthesis by evaluating three generative AI models on semantic label maps derived from driving simulators. It systematically compares two GAN-based I2I methods (Pix2pixHD, OASIS) and a diffusion-based approach (ControlNet) using both Cityscapes and SHIFT-derived labels, across image-quality and perception metrics. The results show GANs excel for Cityscapes-aligned realism, while ControlNet offers fewer artefacts and better structural fidelity when using simulator-generated labels, with diffusion-based methods showing promising stability and generalization. This work demonstrates that diffusion-guided, text- and layout-conditioned generation can be a practical, scalable path to realistic, richly annotated driving datasets, potentially reducing reliance on costly real-world data.

Abstract

Datasets are essential for training and testing vehicle perception algorithms. However, the collection and annotation of real-world images is time-consuming and expensive. Driving simulators offer a solution by automatically generating various driving scenarios with corresponding annotations, but the simulation-to-reality (Sim2Real) domain gap remains a challenge. While most of the Generative Artificial Intelligence (AI) follows the de facto Generative Adversarial Nets (GANs)-based methods, the recent emerging diffusion probabilistic models have not been fully explored in mitigating Sim2Real challenges for driving data synthesis. To explore the performance, this paper applied three different generative AI methods to leverage semantic label maps from a driving simulator as a bridge for the creation of realistic datasets. A comparative analysis of these methods is presented from the perspective of image quality and perception. New synthetic datasets, which include driving images and auto-generated high-quality annotations, are produced with low costs and high scene variability. The experimental results show that although GAN-based methods are adept at generating high-quality images when provided with manually annotated labels, ControlNet produces synthetic datasets with fewer artefacts and more structural fidelity when using simulator-generated labels. This suggests that the diffusion-based approach may provide improved stability and an alternative method for addressing Sim2Real challenges.
Paper Structure (10 sections, 4 figures, 3 tables)

This paper contains 10 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Selected generative AI models.
  5. Evaluation
  6. Experiment Results
  7. Implementation Details
  8. Results and Analysis
  9. Discussion
  10. Conclusion

Figures (4)

  • Figure 1: The general driving data synthesis pipeline. On the left are the images from the simulator, the right are the generated driving datasets. Different generative AI models are explored as synthetic networks. The domain gap can be reduced after the pipeline.
  • Figure 2: The processing pipeline of the different label maps from the CARLA simulator to the Cityscapes style for training. Please note that the picture in (a) is coordinated for better visualisation purposes.
  • Figure 3: Qualitative comparison of results from different generative approaches with the Cityscapes validation set
  • Figure 4: Qualitative comparison of results from different generative approaches with the SHIFT validation set