Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GSAVS: Gaussian Splatting-based Autonomous Vehicle Simulator

Rami Wilson

TL;DR

GSAVS tackles the sim-to-real gap in autonomous driving by representing every scene asset as a 3D Gaussian splat rendered in real time within Unity. By wrapping Gaussian splatting around a classical 3D engine, GSAVS delivers photorealistic environments with easy asset customization, leveraging real-world data (e.g., nuScenes) to build a digital twin of driving scenes. The framework defines a road-spline track to constrain ego motion and uses Gaussian-splat ego and vehicle agents with physics-enabled interactions, enabling diverse training scenarios. Experiments across three task sets show competitive accuracy and favorable resource utilization, highlighting the approach's efficiency and scalability. The work signals a path toward scalable, high-fidelity simulators for robust sim-to-real transfer, with future directions including dynamic Gaussians and relightable assets.

Abstract

Modern autonomous vehicle simulators feature an ever-growing library of assets, including vehicles, buildings, roads, pedestrians, and more. While this level of customization proves beneficial when creating virtual urban environments, this process becomes cumbersome when intending to train within a digital twin or a duplicate of a real scene. Gaussian splatting emerged as a powerful technique in scene reconstruction and novel view synthesis, boasting high fidelity and rendering speeds. In this paper, we introduce GSAVS, an autonomous vehicle simulator that supports the creation and development of autonomous vehicle models. Every asset within the simulator is a 3D Gaussian splat, including the vehicles and the environment. However, the simulator runs within a classical 3D engine, rendering 3D Gaussian splats in real-time. This allows the simulator to utilize the photorealism that 3D Gaussian splatting boasts while providing the customization and ease of use of a classical 3D engine.

GSAVS: Gaussian Splatting-based Autonomous Vehicle Simulator

TL;DR

GSAVS tackles the sim-to-real gap in autonomous driving by representing every scene asset as a 3D Gaussian splat rendered in real time within Unity. By wrapping Gaussian splatting around a classical 3D engine, GSAVS delivers photorealistic environments with easy asset customization, leveraging real-world data (e.g., nuScenes) to build a digital twin of driving scenes. The framework defines a road-spline track to constrain ego motion and uses Gaussian-splat ego and vehicle agents with physics-enabled interactions, enabling diverse training scenarios. Experiments across three task sets show competitive accuracy and favorable resource utilization, highlighting the approach's efficiency and scalability. The work signals a path toward scalable, high-fidelity simulators for robust sim-to-real transfer, with future directions including dynamic Gaussians and relightable assets.

Abstract

Modern autonomous vehicle simulators feature an ever-growing library of assets, including vehicles, buildings, roads, pedestrians, and more. While this level of customization proves beneficial when creating virtual urban environments, this process becomes cumbersome when intending to train within a digital twin or a duplicate of a real scene. Gaussian splatting emerged as a powerful technique in scene reconstruction and novel view synthesis, boasting high fidelity and rendering speeds. In this paper, we introduce GSAVS, an autonomous vehicle simulator that supports the creation and development of autonomous vehicle models. Every asset within the simulator is a 3D Gaussian splat, including the vehicles and the environment. However, the simulator runs within a classical 3D engine, rendering 3D Gaussian splats in real-time. This allows the simulator to utilize the photorealism that 3D Gaussian splatting boasts while providing the customization and ease of use of a classical 3D engine.

Paper Structure

This paper contains 11 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Simulation
  3. Environment
  4. Road Spline Track
  5. Autonomous Driving
  6. Ego Agent
  7. Vehicle Agent
  8. Experiments
  9. Results
  10. Future Works
  11. Conclusion

Figures (5)

  • Figure 1: Environment generation from a 3D Gaussian splat of a scene from the nuScenes dataset nuscenes
  • Figure 2:
  • Figure 3: RoadBlockAssets are offset down by half the height of the ego vehicle so that the virtual front camera observes the point of view of the input cameras.
  • Figure 4: Example of an ego vehicle asset. Attached to the wheels are the wheel colliders that allow for torque and steering vectors to be applied to the vehicle. A rectangular, configurable collision boundary is attached to the ego vehicle to detect collisions with the environment or other vehicles
  • Figure 5: Example of a vehicle agent asset. The vehicle's position is manipulated for each time-step $t$ along the spline A rectangular, configurable collision boundary is attached to the vehicle agent to detect collisions with the ego vehicle.