CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation

TL;DR

CADSim tackles the challenge of creating scalable, high-fidelity vehicle assets for sensor simulation by leveraging CAD priors and differentiable rendering to reconstruct articulated, photorealistic meshes from sparse in-the-wild data. It introduces a CAD-informed mesh representation with a low-dimensional CAD library, integrated into a differentiable energy framework that jointly optimizes geometry, appearance, and sensor poses. Through extensive experiments on MVMC and PandaVehicle, CADSim outperforms NeRF-based and geometry-based baselines in novel-view synthesis and LiDAR simulation and enables effective downstream perception testing and realistic multi-sensor insertion. The approach reduces the simulation-to-real domain gap and enables controllable, texture-transferable asset generation for large-scale autonomy evaluation.

Abstract

Realistic simulation is key to enabling safe and scalable development of % self-driving vehicles. A core component is simulating the sensors so that the entire autonomy system can be tested in simulation. Sensor simulation involves modeling traffic participants, such as vehicles, with high quality appearance and articulated geometry, and rendering them in real time. The self-driving industry has typically employed artists to build these assets. However, this is expensive, slow, and may not reflect reality. Instead, reconstructing assets automatically from sensor data collected in the wild would provide a better path to generating a diverse and large set with good real-world coverage. Nevertheless, current reconstruction approaches struggle on in-the-wild sensor data, due to its sparsity and noise. To tackle these issues, we present CADSim, which combines part-aware object-class priors via a small set of CAD models with differentiable rendering to automatically reconstruct vehicle geometry, including articulated wheels, with high-quality appearance. Our experiments show our method recovers more accurate shapes from sparse data compared to existing approaches. Importantly, it also trains and renders efficiently. We demonstrate our reconstructed vehicles in several applications, including accurate testing of autonomy perception systems.

Figures (27)

• Figure 1: CADSim recovers shape, appearance and illumination in a robust and scalable way from sensor observations. The reconstructed assets are of high fidelity, part-aware, geometry-aligned and compatible with graphics engine, enabling efficient, realistic and controllable simulation.
• Figure 2: Overview of CADSim for 3D Reconstruction on in-the-wild data.
• Figure 3: Vehicle model used in CADSim. For simplicity, we only show the offsets along the axle axis.
• Figure 4: Qualitative results on PandaVehicle for novel view synthesis. Compared to existing reconstruction approaches, CADSim produces more robust and realistic results on large extrapolation.
• Figure 5: Evaluation of NVS on MVMC zhang2021ners.