SimULi: Real-Time LiDAR and Camera Simulation with Unscented Transforms

TL;DR

SimULi introduces real-time, arbitrary-camera and LiDAR rendering by extending 3DGUT with LiDAR support and a factorized, cross-sensor Gaussian representation. It solves cross-sensor inconsistencies with an anchoring strategy and employs automated LiDAR tiling and ray-based culling to achieve 10–20x speedups over ray tracing and 1.5–10x over prior rasterization methods, while matching or surpassing state-of-the-art fidelity on Waymo and PandaSet. The approach models camera and LiDAR data with separate particle sets and optimizes with reconstruction, anchoring, and regularization losses to balance multi-sensor accuracy. The results demonstrate superior performance across camera and LiDAR metrics, with practical implications for safe, scalable autonomous vehicle testing and development.

Abstract

Rigorous testing of autonomous robots, such as self-driving vehicles, is essential to ensure their safety in real-world deployments. This requires building high-fidelity simulators to test scenarios beyond those that can be safely or exhaustively collected in the real-world. Existing neural rendering methods based on NeRF and 3DGS hold promise but suffer from low rendering speeds or can only render pinhole camera models, hindering their suitability to applications that commonly require high-distortion lenses and LiDAR data. Multi-sensor simulation poses additional challenges as existing methods handle cross-sensor inconsistencies by favoring the quality of one modality at the expense of others. To overcome these limitations, we propose SimULi, the first method capable of rendering arbitrary camera models and LiDAR data in real-time. Our method extends 3DGUT, which natively supports complex camera models, with LiDAR support, via an automated tiling strategy for arbitrary spinning LiDAR models and ray-based culling. To address cross-sensor inconsistencies, we design a factorized 3D Gaussian representation and anchoring strategy that reduces mean camera and depth error by up to 40% compared to existing methods. SimULi renders 10-20x faster than ray tracing approaches and 1.5-10x faster than prior rasterization-based work (and handles a wider range of camera models). When evaluated on two widely benchmarked autonomous driving datasets, SimULi matches or exceeds the fidelity of existing state-of-the-art methods across numerous camera and LiDAR metrics.

Figures (10)

• Figure 1: SimULi. We design a factorized 3D Gaussian representation that encodes camera and LiDAR information into separate sets of 3D Gaussians joined via nearest-neighbor anchoring loss (left). To efficiently render LiDAR scans (middle), we extend 3DGUT wu20253dgut with an automated tiling strategy and ray-based culling. When compared to existing methods, we render 1.5-20$\times$ faster and match or exceed LiDAR and camera quality (right) across a wide range of metrics.
• Figure 2: Method Overview. We model the scene as a dynamic graph Ost_2021_CVPR and parameterize the background and each actor with camera and LiDAR 3D Gaussians (left). We render camera views similar to 3DGUT wu20253dgut and derive an automated tiling strategy and ray-based culling to efficiently render LiDAR (middle). We sample an image and LiDAR scan at each training step to optimize our representation (right). To improve camera novel view synthesis with LiDAR-supervised geometry, we anchor camera Gaussians near surfaces via nearest-neighbor loss.
• Figure 3: LiDAR Tiling. As the measurement pattern of commonly used LiDAR sensors pandaset is irregular (left), rendering with equally spaced tiles is highly inefficient. We compute the normalized CDF of elevation angles using a predefined histogram bin count and set elevation tiling boundaries at angles where the CDF values cross integer boundaries (middle). We then compute an azimuth tile count such that the beam count per tile differs at most by 8 samples (right).
• Figure 4: Depth Supervision. Prior work encodes camera and LiDAR into the same representation constrained with a LiDAR-supervised depth loss. As cross-sensor data is not fully consistent, this forces the representation to prioritize camera instead of LiDAR quality (left) or the inverse (middle), as shown by PSNR and chamfer distance. We factorize each modality into its own particle set joined via nearest-neighbor anchoring loss, improving the quality of each (right).
• Figure 5: Static NVS. Projecting LiDAR as a sparse depth map causes inaccuracies that degrade 3DGUT's rendering of the pole (above), which we avoid by rendering LiDAR directly. Our bilateral grids also prevent floaters (middle), allowing us to render slightly faster. SplatAD's CNN does not properly recover the sign lettering (above) or rear car seat (below), while our simpler pipeline does.