Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing Quality Metrics for Neural Reality Gap Input Mitigation in Autonomous Driving Testing

Stefano Carlo Lambertenghi, Andrea Stocco

TL;DR

This paper tackles the sim2real gap in vision-based autonomous driving testing by evaluating pix2pix and CycleGAN as I2I translators that convert simulated driving images into real-like inputs. It systematically analyzes 13 existing evaluation metrics—covering distribution- and single-image, content-based, and perception-based categories—and introduces task-tailored perception metrics to better predict ADS behavior. The study finds that I2I effectiveness is task-dependent, while many standard metrics do not consistently align with ADS performance; however, a task-specific perception metric (OC-TSS) demonstrates robust correlations across tasks, offering a practical tool for selecting appropriate I2I approaches for sim2real mitigation. The findings underscore the importance of semantic-preserving, task-aware evaluation in ADS testing and point to future work with diffusion-based generation and broader, system-level validations to enable reliable real-world deployment.

Abstract

Simulation-based testing of automated driving systems (ADS) is the industry standard, being a controlled, safe, and cost-effective alternative to real-world testing. Despite these advantages, virtual simulations often fail to accurately replicate real-world conditions like image fidelity, texture representation, and environmental accuracy. This can lead to significant differences in ADS behavior between simulated and real-world domains, a phenomenon known as the sim2real gap. Researchers have used Image-to-Image (I2I) neural translation to mitigate the sim2real gap, enhancing the realism of simulated environments by transforming synthetic data into more authentic representations of real-world conditions. However, while promising, these techniques may potentially introduce artifacts, distortions, or inconsistencies in the generated data that can affect the effectiveness of ADS testing. In our empirical study, we investigated how the quality of image-to-image (I2I) techniques influences the mitigation of the sim2real gap, using a set of established metrics from the literature. We evaluated two popular generative I2I architectures, pix2pix, and CycleGAN, across two ADS perception tasks at a model level, namely vehicle detection and end-to-end lane keeping, using paired simulated and real-world datasets. Our findings reveal that the effectiveness of I2I architectures varies across different ADS tasks, and existing evaluation metrics do not consistently align with the ADS behavior. Thus, we conducted task-specific fine-tuning of perception metrics, which yielded a stronger correlation. Our findings indicate that a perception metric that incorporates semantic elements, tailored to each task, can facilitate selecting the most appropriate I2I technique for a reliable assessment of the sim2real gap mitigation.

Assessing Quality Metrics for Neural Reality Gap Input Mitigation in Autonomous Driving Testing

TL;DR

This paper tackles the sim2real gap in vision-based autonomous driving testing by evaluating pix2pix and CycleGAN as I2I translators that convert simulated driving images into real-like inputs. It systematically analyzes 13 existing evaluation metrics—covering distribution- and single-image, content-based, and perception-based categories—and introduces task-tailored perception metrics to better predict ADS behavior. The study finds that I2I effectiveness is task-dependent, while many standard metrics do not consistently align with ADS performance; however, a task-specific perception metric (OC-TSS) demonstrates robust correlations across tasks, offering a practical tool for selecting appropriate I2I approaches for sim2real mitigation. The findings underscore the importance of semantic-preserving, task-aware evaluation in ADS testing and point to future work with diffusion-based generation and broader, system-level validations to enable reliable real-world deployment.

Abstract

Simulation-based testing of automated driving systems (ADS) is the industry standard, being a controlled, safe, and cost-effective alternative to real-world testing. Despite these advantages, virtual simulations often fail to accurately replicate real-world conditions like image fidelity, texture representation, and environmental accuracy. This can lead to significant differences in ADS behavior between simulated and real-world domains, a phenomenon known as the sim2real gap. Researchers have used Image-to-Image (I2I) neural translation to mitigate the sim2real gap, enhancing the realism of simulated environments by transforming synthetic data into more authentic representations of real-world conditions. However, while promising, these techniques may potentially introduce artifacts, distortions, or inconsistencies in the generated data that can affect the effectiveness of ADS testing. In our empirical study, we investigated how the quality of image-to-image (I2I) techniques influences the mitigation of the sim2real gap, using a set of established metrics from the literature. We evaluated two popular generative I2I architectures, pix2pix, and CycleGAN, across two ADS perception tasks at a model level, namely vehicle detection and end-to-end lane keeping, using paired simulated and real-world datasets. Our findings reveal that the effectiveness of I2I architectures varies across different ADS tasks, and existing evaluation metrics do not consistently align with the ADS behavior. Thus, we conducted task-specific fine-tuning of perception metrics, which yielded a stronger correlation. Our findings indicate that a perception metric that incorporates semantic elements, tailored to each task, can facilitate selecting the most appropriate I2I technique for a reliable assessment of the sim2real gap mitigation.
Paper Structure (24 sections, 3 figures, 4 tables)

This paper contains 24 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Image-to-Image Translation Models
  3. Image-to-Image Models Evaluation Metrics
  4. Distribution-level Metrics
  5. Single-Image Metrics
  6. Content-based Single-Image Metrics
  7. Perception-based Single-Image Metrics
  8. Empirical Study
  9. Research Questions
  10. Datasets
  11. Vehicle Detection Dataset
  12. Lane Keeping Dataset
  13. ADS Under Test
  14. Vehicle Detection
  15. Lane Keeping
  16. ...and 9 more sections

Figures (3)

  • Figure 1: An example of dataset sim2real pairs for the Vehicle Detection (a-b) and Lane Keeping (c-d) datasets.
  • Figure 2: Distributions of ADS behavior on different image domains. The black horizontal line represents the median of the distribution, whereas the color indicates whether the median is lower (green) or same/higher (red) as compared to the simulated domain (yellow); In each boxplot, distributions with lower values are preferable. The figure is best viewed in color.
  • Figure 3: Translation quality comparison: P1 (low-quality pix2pix) and P3 (high-quality pix2pix). Rows depict simulated/reference images, neural-generated images by P1 and P3, ground truth and vehicle detection ground truth and output (green and red boxes) with confidence (in percentage), ADS attention mask, and TSS/OC-TSS values---larger values of the metrics highlight the error between the semantic representation of the original and generated images, thus characterizing neural translation effectiveness.