Mirage: One-Step Video Diffusion for Photorealistic and Coherent Asset Editing in Driving Scenes

TL;DR

Mirage tackles the challenge of photorealistic, temporally coherent 3D asset editing in driving scenes by introducing a one-step video diffusion framework built on a pretrained text-to-video model. It solves spatial detail loss in 3D VAEs with temporally agnostic latent injection of 2D mid-latents and cross-modal fusion, while preserving causal temporal structure via 3D LoRA adapters. A two-stage data processing pipeline, MirageDrive, aligns scene and asset Gaussians to yield clean supervision for VAE adaptation and harmonization training. Extensive quantitative and qualitative results show Mirage outperforms baselines in realism and temporal stability, and ablation studies validate the design choices. The approach generalizes to other video-to-video tasks and provides a strong baseline for future controllable, high-fidelity video editing in autonomous driving contexts.

Abstract

Vision-centric autonomous driving systems rely on diverse and scalable training data to achieve robust performance. While video object editing offers a promising path for data augmentation, existing methods often struggle to maintain both high visual fidelity and temporal coherence. In this work, we propose \textbf{Mirage}, a one-step video diffusion model for photorealistic and coherent asset editing in driving scenes. Mirage builds upon a text-to-video diffusion prior to ensure temporal consistency across frames. However, 3D causal variational autoencoders often suffer from degraded spatial fidelity due to compression, and directly passing 3D encoder features to decoder layers breaks temporal causality. To address this, we inject temporally agnostic latents from a pretrained 2D encoder into the 3D decoder to restore detail while preserving causal structures. Furthermore, because scene objects and inserted assets are optimized under different objectives, their Gaussians exhibit a distribution mismatch that leads to pose misalignment. To mitigate this, we introduce a two-stage data alignment strategy combining coarse 3D alignment and fine 2D refinement, thereby improving alignment and providing cleaner supervision. Extensive experiments demonstrate that Mirage achieves high realism and temporal consistency across diverse editing scenarios. Beyond asset editing, Mirage can also generalize to other video-to-video translation tasks, serving as a reliable baseline for future research. Our code is available at https://github.com/wm-research/mirage.

Figures (8)

• Figure 1: Photorealistic results of Mirage. GPT-4V visual question answering reveals three key properties: (i) perceptual realism, with no detected inconsistencies; (ii) behavioral out-of-distribution effects, which lead to incorrect reasoning; and (iii) temporal consistency, with full video results provided in the supplementary material. Right: Comparison of naïve insertion, Mirage results, and ground truth.
• Figure 2: Overview of the framework and training flow of Mirage. Given a naïve insertion video, Mirage encodes the input using both a 2D VAE encoder and a causal 3D VAE encoder. The 2D intermediate latents are fused with 3D latents through cross-modal fusion blocks. A diffusion transformer, equipped with 2D LoRA adapters, refines the NI latent toward the ground-truth (GT) latent. The 3D VAE decoder then reconstructs the edited video using two sets of LoRA modules: Reconstruction-LoRA for VAE adaptation and Harmonization-LoRA for harmonization fine-tuning.
• Figure 3: Qualitative comparison of asset insertion results. Naïve insertion produces visible artifacts and inconsistent shading, while R3D2 partially improves realism but still shows lighting and geometry mismatches. Mirage generates photorealistic, temporally consistent vehicles that closely match ground-truth appearance.
• Figure 4: Vehicle identity preservation comparison. We retrain R3D2 on the MirageDrive dataset to ensure consistent geometry before evaluating temporal behavior. Even under this controlled setting, R3D2 shows clear appearance drift across distant frames. In contrast, Mirage preserves stable shape, reflectance, and vehicle identity, remaining closely aligned with the ground truth.
• Figure 5: Qualitative examples from the R3D3 dataset. Horizontal and vertical reference lines, together with arrows, are used to highlight geometric misalignment between the naïvely inserted assets and the ground-truth objects. All frames are taken from the R3D2 r3d2 paper (Supp. E).