Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Instance-Level Safety-Aware Fidelity of Synthetic Data and Its Calibration

Chih-Hong Cheng, Paul Stöckel, Xingyu Zhao

TL;DR

An optimization method is suggested to refine the synthetic data generator, reducing fidelity gaps identified by deep learning components, and experiments show this tuning enhances the correlation between safety-critical errors in synthetic and real data.

Abstract

Modeling and calibrating the fidelity of synthetic data is paramount in shaping the future of safe and reliable self-driving technology by offering a cost-effective and scalable alternative to real-world data collection. We focus on its role in safety-critical applications, introducing four types of instance-level fidelity that go beyond mere visual input characteristics. The aim is to ensure that applying testing on synthetic data can reveal real-world safety issues, and the absence of safety-critical issues when testing under synthetic data can provide a strong safety guarantee in real-world behavior. We suggest an optimization method to refine the synthetic data generator, reducing fidelity gaps identified by deep learning components. Experiments show this tuning enhances the correlation between safety-critical errors in synthetic and real data.

Instance-Level Safety-Aware Fidelity of Synthetic Data and Its Calibration

TL;DR

An optimization method is suggested to refine the synthetic data generator, reducing fidelity gaps identified by deep learning components, and experiments show this tuning enhances the correlation between safety-critical errors in synthetic and real data.

Abstract

Modeling and calibrating the fidelity of synthetic data is paramount in shaping the future of safe and reliable self-driving technology by offering a cost-effective and scalable alternative to real-world data collection. We focus on its role in safety-critical applications, introducing four types of instance-level fidelity that go beyond mere visual input characteristics. The aim is to ensure that applying testing on synthetic data can reveal real-world safety issues, and the absence of safety-critical issues when testing under synthetic data can provide a strong safety guarantee in real-world behavior. We suggest an optimization method to refine the synthetic data generator, reducing fidelity gaps identified by deep learning components. Experiments show this tuning enhances the correlation between safety-critical errors in synthetic and real data.
Paper Structure (20 sections, 1 theorem, 11 equations, 4 figures, 1 table)

This paper contains 20 sections, 1 theorem, 11 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Foundations
  4. The Facets of Instance-level Fidelity
  5. Input, output, and latent feature fidelity
  6. Safety-aware fidelity
  7. Calibrating Data Generation for Safety-aware Fidelity
  8. Evaluation
  9. Research questions
  10. RQ1 (assessment): Can the proposed SA-fidelity metric effectively differentiate and rank different synthetic data generators in terms of safety?
  11. RQ2 (calibration): For a given synthetic data generator, can SA-fidelity be utilized for calibrating optimal configurations geared toward safety?
  12. RQ3 (integration with scenario-based testing): When integrating SA-fidelity concepts into the engineering process of scenario-based virtual testing, what are additional challenges to be considered?
  13. Experiment setup
  14. Synthetic data generation
  15. DNNs under testing
  16. ...and 5 more sections

Key Result

Lemma 1

Given a synthetic data point $x^s = g_{\theta}(sd)$, assume the perception module $f$ is locally robust with a maximum robustness bound $\epsilon_{r}$, i.e., then if a synthetic data point $x^s$ is IV-fidelitous under $\langle \mathcal{D}_{in},\epsilon_{in} \rangle$ where $\epsilon_{in} \leq \epsilon_{r}$, $x^s$ is also OV-fidelitous under $\langle \mathcal{D}_{out},\epsilon_{out} \rangle$.

Figures (4)

  • Figure 1: The spectrum of instance-level fidelity from the perspectives of DNN inputs, features extracted, DNN outputs and its effect on system safety.
  • Figure 2: Example of a real-world KITTI image (a), and its synthetic images generated via "style transfer" (b-d) and "scenario understanding" (e).
  • Figure 3: Examples of inconsistent object detection on safety-relevant objects in real vs. synthetic images.
  • Figure 4: Statistics of inconsistent predictions, over 3 DNNs and 3 synthetic datasets.

Theorems & Definitions (16)

  • Remark 1: Approximation of $\emph{{\smallRW}}(sd)$
  • Definition 1: IV-fidelity
  • Definition 2: OV-fidelity
  • Remark 2: Conservative bound on distance-based fidelity
  • Lemma 1: IV-fidelity, robustness, and OV-fidelity
  • proof
  • Remark 3: Photo-realistic is not real
  • Definition 3: LF-fidelity
  • Example 1
  • Example 1: continued
  • ...and 6 more