OASim: an Open and Adaptive Simulator based on Neural Rendering for Autonomous Driving

TL;DR

The paper presents OASim, an open-source simulator for autonomous driving that uses neural implicit reconstruction and rendering (NeRF and 3D Gaussian Splatting) to generate high-fidelity, customizable data. It introduces a four-layer architecture and a workflow that converts real-world multi-sensor data into trainable neural fields, with a foreground asset library for dynamic objects and an interactive interface for trajectory and sensor edits. Key contributions include the integration of StreetSurf-based background reconstruction, automatic annotation, and closed-loop data generation to cover long-tail and edge cases. The approach aims to reduce data collection costs and safety risks while enabling scalable, high-quality data generation for perception, planning, and control tasks.

Abstract

With deep learning and computer vision technology development, autonomous driving provides new solutions to improve traffic safety and efficiency. The importance of building high-quality datasets is self-evident, especially with the rise of end-to-end autonomous driving algorithms in recent years. Data plays a core role in the algorithm closed-loop system. However, collecting real-world data is expensive, time-consuming, and unsafe. With the development of implicit rendering technology and in-depth research on using generative models to produce data at scale, we propose OASim, an open and adaptive simulator and autonomous driving data generator based on implicit neural rendering. It has the following characteristics: (1) High-quality scene reconstruction through neural implicit surface reconstruction technology. (2) Trajectory editing of the ego vehicle and participating vehicles. (3) Rich vehicle model library that can be freely selected and inserted into the scene. (4) Rich sensors model library where you can select specified sensors to generate data. (5) A highly customizable data generation system can generate data according to user needs. We demonstrate the high quality and fidelity of the generated data through perception performance evaluation on the Carla simulator and real-world data acquisition. Code is available at https://github.com/PJLab-ADG/OASim.

Figures (9)

• Figure 1: Workflow of OASim. Its hierarchical structure can be divided into four layers, including data layer, back-end layer, front-end layer and application layer.
• Figure 2: OASim interface.
• Figure 3: Sensor editing and rendering interface.
• Figure 4: Qualitative image rendering results.
• Figure 5: Non-rigid pedestrian rendering results.