Boosting Visual Fidelity in Driving Simulations through Diffusion Models

TL;DR

The paper addresses the challenge of low physical validity in driving simulations due to limited visual fidelity. It introduces DRIVE, a diffusion-model-based pipeline that performs real-time style transfer to render photorealistic views, leveraging ControlNet conditioning, LoRa adapters, Latent Consistency Models, and ROS 2 for cross-machine deployment. The authors present a detailed system design focusing on acceleration, within-frame and cross-frame consistency, and a preliminary user study indicating improved visual realism with DRIVE, though task performance evidence remains inconclusive. The work demonstrates a scalable, data-driven path to enhance realism in driving simulations and outlines practical guidelines and future directions, including image harmonization in mixed reality and interactive conditioning with human feedback.

Abstract

Diffusion models have made substantial progress in facilitating image generation and editing. As the technology matures, we see its potential in the context of driving simulations to enhance the simulated experience. In this paper, we explore this potential through the introduction of a novel system designed to boost visual fidelity. Our system, DRIVE (Diffusion-based Realism Improvement for Virtual Environments), leverages a diffusion model pipeline to give a simulated environment a photorealistic view, with the flexibility to be adapted for other applications. We conducted a preliminary user study to assess the system's effectiveness in rendering realistic visuals and supporting participants in performing driving tasks. Our work not only lays the groundwork for future research on the integration of diffusion models in driving simulations but also provides practical guidelines and best practices for their application in this context.

Figures (10)

• Figure 1: A system diagram for the DRIVE system.
• Figure 2: Diffusion model pipeline that gives our simulation a photorealistic view. ControlNet was used to ensure the generative process does not alter the lane marking on the road. LoRa adapters are used to perform style transfer and incorporate latent consistent models. The pipeline is currently running at 10 fps for an input image of resolution 640x480. (For illustration, the images shown above are rendered at a resolution of 512 x 512 with 5 inferencing steps.)
• Figure 3: An overview of images in the racetrack dataset for LoRa adapter training.
• Figure 4: Simulated racetrack environment for our preliminary user study. The leftmost image is the input to the DRIVE system, which is what the participant will see in condition B. The rightmost image is the output image from the DRIVE system, which is what the participant will see in condition A. Since we prioritize inference speed, the diffusion pipeline only takes one inference step for every input image to perform style transfer.
• Figure 5: The hardware setup for our user study.