Sim2Real Diffusion: Leveraging Foundation Vision Language Models for Adaptive Automated Driving

TL;DR

The paper tackles the perceptual sim2real gap in vision-based autonomous driving by introducing a conditional latent diffusion framework that learns cross-domain representations using text and image prompts. It decouples domain adaptation from the core autonomy stack, enabling rapid, few-shot fine-tuning and modular handling of multiple domain shifts. Through extensive ablations, performance benchmarks, and two behavioral cloning case studies, the approach demonstrates around a 40% reduction in perceptual disparity between simulated and real-world data and shows practical real-time capability on varied hardware. This work provides a flexible, scalable path to bridge simulation and reality for ADS, with strong implications for rapid deployment and broader domain coverage.

Abstract

Simulation-based design, optimization, and validation of autonomous vehicles have proven to be crucial for their improvement over the years. Nevertheless, the ultimate measure of effectiveness is their successful transition from simulation to reality (sim2real). However, existing sim2real transfer methods struggle to address the autonomy-oriented requirements of balancing: (i) conditioned domain adaptation, (ii) robust performance with limited examples, (iii) modularity in handling multiple domain representations, and (iv) real-time performance. To alleviate these pain points, we present a unified framework for learning cross-domain adaptive representations through conditional latent diffusion for sim2real transferable automated driving. Our framework offers options to leverage: (i) alternate foundation models, (ii) a few-shot fine-tuning pipeline, and (iii) textual as well as image prompts for mapping across given source and target domains. It is also capable of generating diverse high-quality samples when diffusing across parameter spaces such as times of day, weather conditions, seasons, and operational design domains. We systematically analyze the presented framework and report our findings in terms of performance benchmarks and ablation studies. Additionally, we demonstrate its serviceability for autonomous driving using behavioral cloning case studies. Our experiments indicate that the proposed framework is capable of bridging the perceptual sim2real gap by over 40%.

Figures (7)

• Figure 1: Proposed sim2real diffusion approach depicting the vehicle performing cross-domain perceptual adaptation for transferring simulation-trained algorithms to reality.
• Figure 2: Proposed framework for enabling sim2real transfer of vision-based autonomy algorithms through the learning of adaptive cross-domain representations using image and text conditioning within a latent diffusion architecture.
• Figure 3: Results of the 1$^\textrm{st}$ ablation study performed on (a) sim and (b) real camera frames to assess the effect of domain adaptation direction {(c) real2sim, (d) sim2real}, foundation model {(e) SDXL, (f) SDXL-Turbo, (g) SD3M}, denoising steps {(h) 1, (i) 5, (j) 30}, and input image resolution {(k) 2560×1096 px, (l) 640×274 px}.
• Figure 4: Results of the 2$^\textrm{nd}$ ablation study performed on model architecture with a common (a) image prompt (if applicable) and (b) input image, to assess the effect of image-prompting {(c, d) no, (e, f) yes} and fine-tuning {(c, e) no, (d, f) yes}.
• Figure 5: Results of the 3$^\textrm{rd}$ ablation study performed on fine-tuned model with pure text conditioning to assess output sample diversity: <autodrive_small_onroad> (a) night, (b) rain, (c) fog, (d) fall, (e) winter, (f) spring, (g) racetrack; <autodrive_small_racing> (h) sunrise, (i) desert, (j) public road; <autodrive_large_offroad> (k) snow, (l) fall.