Unified Sensor Simulation for Autonomous Driving

TL;DR

This work introduces XSIM, a sensor simulation framework for autonomous driving that extends 3DGUT splatting with a generalized rolling-shutter modeling tailored for autonomous driving applications, and introduces an extended 3D Gaussian representation that incorporates two distinct opacity parameters to resolve mismatches between geometry and color distributions.

Abstract

In this work, we introduce \textbf{XSIM}, a sensor simulation framework for autonomous driving. XSIM extends 3DGUT splatting with a generalized rolling-shutter modeling tailored for autonomous driving applications. Our framework provides a unified and flexible formulation for appearance and geometric sensor modeling, enabling rendering of complex sensor distortions in dynamic environments. We identify spherical cameras, such as LiDARs, as a critical edge case for existing 3DGUT splatting due to cyclic projection and time discontinuities at azimuth boundaries leading to incorrect particle projection. To address this issue, we propose a phase modeling mechanism that explicitly accounts temporal and shape discontinuities of Gaussians projected by the Unscented Transform at azimuth borders. In addition, we introduce an extended 3D Gaussian representation that incorporates two distinct opacity parameters to resolve mismatches between geometry and color distributions. As a result, our framework provides enhanced scene representations with improved geometric consistency and photorealistic appearance. We evaluate our framework extensively on multiple autonomous driving datasets, including Waymo Open Dataset, Argoverse 2, and PandaSet. Our framework consistently outperforms strong recent baselines and achieves state-of-the-art performance across all datasets. The source code is publicly available at \href{https://github.com/whesense/XSIM}{https://github.com/whesense/XSIM}.

Figures (11)

• Figure 1: Synthetic example of LiDAR rendering. In regions near the azimuth discontinuity border, the standard 3DGUT projection results in partially missing and distorted range image renders. Our phase modeling approach alleviates this issue and also effectively handles surfaces observed twice due to the rolling shutter.
• Figure 2: Novel view synthesis (Lane shift 3m) Qualitative comparison on Waymo Open Dataset demonstrates that XSIM provides scene representation which can be consistently rendered from novel ego-vehicle trajectories.
• Figure 3: Depth map rendering. Qualitative comparison of depth map rendering on Waymo Open Dataset. Compared to previous methods, our framework provides smooth geometric representations with high level of details.
• Figure 4: LiDAR discontinuities occurring near azimuth border. a) Even when sensor is stationary and covers exactly 360 degrees, time discontinuity due to the rolling shutter may lead to objects observed twice. b) Ego movement in combination with rolling shutter leads to spatial discontinuities
• Figure 5: Modeling LiDAR opacity (LO) separately resolves geometry and color distributions mismatch, and increases quality of appearance modeling for translucent surfaces and specular reflections.