Quantifying the Sim2real Gap for GPS and IMU Sensors

TL;DR

This paper demonstrates that by using a state of the art state-estimation package as a ``judge'', and by evaluating the performance of this state-estimator in both real and simulated scenarios, it can isolate the sim2real discrepancies stemming from sensor simulations alone.

Abstract

Simulation can and should play a critical role in the development and testing of algorithms for autonomous agents. What might reduce its impact is the ``sim2real'' gap -- the algorithm response differs between operation in simulated versus real-world environments. This paper introduces an approach to evaluate this gap, focusing on the accuracy of sensor simulation -- specifically IMU and GPS -- in velocity estimation tasks for autonomous agents. Using a scaled autonomous vehicle, we conduct 40 real-world experiments across diverse environments then replicate the experiments in simulation with five distinct sensor noise models. We note that direct comparison of raw simulation and real sensor data fails to quantify the sim2real gap for robotics applications. We demonstrate that by using a state of the art state-estimation package as a ``judge'', and by evaluating the performance of this state-estimator in both real and simulated scenarios, we can isolate the sim2real discrepancies stemming from sensor simulations alone. The dataset generated is open-source and publicly available for unfettered use.

Figures (6)

• Figure 1: The state estimator serves as the "judge," processing both real-world and simulated sensor data to produce velocity estimates $\mathbf{v}_{re}$ and $\mathbf{v}_{se}$, respectively. In the real world, ground-truth velocity ($\mathbf{v}_{rg}$) is provided by a validated INS system, while the simulator yields $\mathbf{v}_{sg}$. RMSE values ($E_r$ and $E_s$) are calculated for both the real and simulated cases (Eq. \ref{['eq:RMSE']}). Additionally, Fast Fourier Transforms (FFTs) are employed to compute Wiener Entropy values $S_r$ and $S_s$ (Eq. \ref{['eq:ent_diff']}). This process is repeated across K tests, generating sets $A_1$, $A_2$, $B_1$, and $B_2$. The Wasserstein distance $W_1(A_1,B_1)$ and $W_2(A_2,B_2)$ compares these sets, yielding the Velocity Estimation Performance Difference (VEPD) score as an average.
• Figure 2: ART -- the 1/6 scale vehicle (see 0.2 m vernier caliper for scale) used for real-world experiments.
• Figure 3: Comparison of the distributions of RMSE and Wiener entropy for velocity estimates obtained from different simulated GPS sensor models. Lower RMSE and Wiener entropy values indicate a closer match between simulation and reality. The Ch:RW model (RW stands for Random Walk) demonstrates the most similar Wiener Entropy distribution to the real-world data. Conversely, the AirSim model, with minimal modeled noise, achieves the closest RMSE distribution to reality.
• Figure 4: Using the simulated IMU with real GPS produces a velocity estimate (left orange) that closely resembles the setup when a real IMU is used with a real GPS (right orange). The RMSE (Left: $0.2004$, Right: $0.2083$) and Wiener Entropy (Left: $0.0311$, Right: $0.0272$) are close, which yield a small VEPD.
• Figure 5: The velocity estimates (orange) produced by the state estimator across three varying environments for one out of the ten tests. Left: In simulation with the Ch:RW GPS noise model, Center: In reality using real sensors on grass, Right: In reality using real sensors on inclined concrete. Although the plots show varying velocity profiles due to varying environments, the VEPD score of the sensor model remains the same.