MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Driving

TL;DR

This work introduces MARS, an open-source, NeRF-based autonomous driving simulator that separates foreground instances from the background into modular, instance-aware components. By enabling flexible integration of multiple NeRF backbones, sampling strategies, and multi-modal inputs, it achieves high photo-realism while supporting instance editing and dynamic scene manipulation. The approach delivers state-of-the-art rendering performance on public benchmarks and demonstrates practical benefits from its modular design, including ablations and flexible scene editing. Although training is time-intensive and real-time rendering is not yet feasible, MARS provides a versatile platform for photorealistic driving simulation and future development of NeRF-based autonomous-driving tools.

Abstract

Nowadays, autonomous cars can drive smoothly in ordinary cases, and it is widely recognized that realistic sensor simulation will play a critical role in solving remaining corner cases by simulating them. To this end, we propose an autonomous driving simulator based upon neural radiance fields (NeRFs). Compared with existing works, ours has three notable features: (1) Instance-aware. Our simulator models the foreground instances and background environments separately with independent networks so that the static (e.g., size and appearance) and dynamic (e.g., trajectory) properties of instances can be controlled separately. (2) Modular. Our simulator allows flexible switching between different modern NeRF-related backbones, sampling strategies, input modalities, etc. We expect this modular design to boost academic progress and industrial deployment of NeRF-based autonomous driving simulation. (3) Realistic. Our simulator set new state-of-the-art photo-realism results given the best module selection. Our simulator will be open-sourced while most of our counterparts are not. Project page: https://open-air-sun.github.io/mars/.

Figures (10)

• Figure 1: Pipeline. Left: We first calculate the ray-box intersection of the queried ray $\textbf{r}$ and all visible instance bounding boxes $\{\mathcal{B}_{ij}\}$. For the background node, we directly use the selected scene representation model and the chosen sampler to infer point-wise properties, as in conventional NeRFs. For the foreground nodes, the ray is first transformed into the instance frame as $\textbf{r}_o$ before being processed through foreground node representations (Sec. \ref{['sec:scene-representation']}). Right: All the samples are composed and rendered into RGB images, depth maps, and semantics (Sec. \ref{['sec:rendering']}).
• Figure 2: Illustration on the compositional rendering. Some of the static vehicles in the far region are considered as background objects.
• Figure 3: Visual demonstration on our conflict-free sampling process. We use uniform sampling in all nodes for illustration.
• Figure 4: We show that the background truncated samples cause background-foreground ambiguity without our regularization.
• Figure 5: Qualitative image reconstruction results on KITTI dataset.