Instance Performance Difference: A Metric to Measure the Sim-To-Real Gap in Camera Simulation

TL;DR

The concept of Instance Performance Difference (IPD) is introduced, a metric designed to measure the gap in performance that a robotics perception task experiences when working with real vs. synthetic pictures, and it is demonstrated that this supports robust sim-to-real transfer for perception algorithms in real-world robotics applications.

Abstract

In this contribution, we introduce the concept of Instance Performance Difference (IPD), a metric designed to measure the gap in performance that a robotics perception task experiences when working with real vs. synthetic pictures. By pairing synthetic and real instances in the pictures and evaluating their performance similarity using perception algorithms, IPD provides a targeted metric that closely aligns with the needs of real-world applications. We explain and demonstrate this metric through a rock detection task in lunar terrain images, highlighting the IPD's effectiveness in identifying the most realistic image synthesis method. The metric is thus instrumental in creating synthetic image datasets that perform in perception tasks like real-world photo counterparts. In turn, this supports robust sim-to-real transfer for perception algorithms in real-world robotics applications.

Figures (3)

• Figure 1: Indexed pairs of rocks from the (left) real and (right) synthetic pictures, respectively. The Pink boxes are the ground-truth labels with annotated rock indices, and the red boxes are the predicted labels with annotated rock indices and prediction confidence.
• Figure 2: The point-set registration problem before and after solved of some cases shown for demonstration. (GT: ground-true, bbox: bounding box, synth: synthetic)
• Figure 3: Illustration of IPD. The performance task trained on Domain A is treated as a mapping that projects from the image domains to the performance-value domains, which is only effective in Domain A.