MiLA: Multi-view Intensive-fidelity Long-term Video Generation World Model for Autonomous Driving

TL;DR

MiLA tackles the challenge of producing high-fidelity, long-duration driving videos by learning a world-model that is controllable via simple waypoints and multi-view cues. It introduces a coarse-to-(Re)fine generation framework complemented by a Joint Denoising and Correcting Flow and a Temporal Progressive Denoising Scheduler to maintain fidelity and temporal coherence over long sequences. Through extensive nuScenes experiments, MiLA achieves state-of-the-art quality in both front-view and multi-view video generation and demonstrates strong ablations showing the value of its components. The work advances data-efficient autonomous driving training by enabling scalable synthesis of long, annotated driving scenes, with implications for downstream perception and planning models.

Abstract

In recent years, data-driven techniques have greatly advanced autonomous driving systems, but the need for rare and diverse training data remains a challenge, requiring significant investment in equipment and labor. World models, which predict and generate future environmental states, offer a promising solution by synthesizing annotated video data for training. However, existing methods struggle to generate long, consistent videos without accumulating errors, especially in dynamic scenes. To address this, we propose MiLA, a novel framework for generating high-fidelity, long-duration videos up to one minute. MiLA utilizes a Coarse-to-Re(fine) approach to both stabilize video generation and correct distortion of dynamic objects. Additionally, we introduce a Temporal Progressive Denoising Scheduler and Joint Denoising and Correcting Flow modules to improve the quality of generated videos. Extensive experiments on the nuScenes dataset show that MiLA achieves state-of-the-art performance in video generation quality. For more information, visit the project website: https://github.com/xiaomi-mlab/mila.github.io.

Figures (12)

• Figure 1: An illustration of the existing Recurrent and the existing Divide-and-Conquer generation frameworks. Left: The generation of newly predicted frames always conditioned the previously generated frames recursively with an LDM vistaGaia-1. Right: Anchor frames are firstly drawn by an LDM, and frames between anchor frames are interpolated in parallel with an Interpolation Model videoldmnuwa.
• Figure 2: The overall pipeline of the proposed MiLA. Top: The Coarse-to-(Re)fine generation framework. The conditions of the current (Re)fine stage involve both the corrected high FPS frames from the previous (Re)fine stage and the predicted low FPS anchor frames from the Coarse stage. The low FPS anchor frames are jointly denoised with high FPS interpolation frames to correct the artifacts. Bottom: The generation model structure of MiLA. The inputs of the model include multi-view video (condition frames, noise, and anchor frames occasionally), FPS, waypoints, camera parameters, and brief textual scene descriptions. After encoding multi-modal inputs with different encoders, a DiT-based framework denoises the noised video tokens for $N$ times and outputs the prediction video.
• Figure 3: (a): The predicted frames of two distinct views were generated in increasing temporal order, where frames of the Front View are placed in the first row and frames of the Back Left View are placed in the second row, respectively. All frames are produced using a single diffusion step. Frames that are closer to the condition frames are less degradative. (b): A sample visualization of the function used to compute the noise scale, based on the noisy frame index and denoising timestamps as variables.
• Figure 4: Comparison between Recurrent videoldmnuwa Divide-to-Conquer vistaGaia-1 and the proposed Coarse-to-(Re)fine generation frameworks with two distinct views (front-right and back-right) and three frames (76th, 90th and 114th). All frames are generated with our MiLA and different paradigms for long video generation are adopted. Left: The condition frames for video generation. The predicted videos are placed to Top Right: Recurrent, Middle Right: Divide-and-Conquer and Bottom Right: Coarse-to-(Re)fine.
• Figure 5: Multiple view generation. Columns from left to right record the frames of (a: ) Zoom in visualization, (b: ) Vista, (c: ) Ground Truth of Front Left and Front view and (d: ) our MiLA, respectively.