MAD: Motion Appearance Decoupling for efficient Driving World Models

TL;DR

MAD introduces a decoupled, two-stage framework that converts general-purpose video foundation models into controllable driving world models by first forecasting motion with a Motion Forecaster and then rendering appearance with an Appearance Synthesizer, using a pose-based intermediate representation acquired via pose extractors. By reusing a single backbone through lightweight LoRA adapters and conditioning via the base model's VAE, MAD achieves high efficiency and substantially reduces data/compute requirements for domain adaptation. The authors demonstrate MAD-SVD and MAD-LTX, with MAD-LTX achieving state-of-the-art or near-state-of-the-art generation quality while supporting multiple control modalities and offering open-source availability. Across extensive experiments on OpenDV, human evaluations show consistent quality gains over open baselines, with MAD-LTX achieving faster inference than comparable open models and competitive performance relative to closed proprietary systems. This approach lowers barriers to building high-fidelity driving world models and enables scalable, controllable simulation for autonomous driving research and development.

Abstract

Recent video diffusion models generate photorealistic, temporally coherent videos, yet they fall short as reliable world models for autonomous driving, where structured motion and physically consistent interactions are essential. Adapting these generalist video models to driving domains has shown promise but typically requires massive domain-specific data and costly fine-tuning. We propose an efficient adaptation framework that converts generalist video diffusion models into controllable driving world models with minimal supervision. The key idea is to decouple motion learning from appearance synthesis. First, the model is adapted to predict structured motion in a simplified form: videos of skeletonized agents and scene elements, focusing learning on physical and social plausibility. Then, the same backbone is reused to synthesize realistic RGB videos conditioned on these motion sequences, effectively "dressing" the motion with texture and lighting. This two-stage process mirrors a reasoning-rendering paradigm: first infer dynamics, then render appearance. Our experiments show this decoupled approach is exceptionally efficient: adapting SVD, we match prior SOTA models with less than 6% of their compute. Scaling to LTX, our MAD-LTX model outperforms all open-source competitors, and supports a comprehensive suite of text, ego, and object controls. Project page: https://vita-epfl.github.io/MAD-World-Model/

Figures (12)

• Figure 1: (Top) Our decoupled two-stage pipeline: a Motion Forecaster first generates an abstract intermediate pose representation, which is then used by an Appearance Synthesizer to render the final, photorealistic video. (Bottom) Our method (MAD-LTX) achieves a state-of-the-art quality while requiring a fraction of the training compute budget compared to prior SOTA driving models.
• Figure 2: Training pipeline of our two models. a) depicts how the pose video is noised in the diffusion process and given as input to the LoRA of the base DiT model, namely $F_\theta$, together with different conditions such as text captions, first frame in RGB and optionally, our visual representation of the ego motion and other objects location. The denoised pose tokens can be decoded back to the generated pose video at inference time. Similarly, b) shows the training scheme for our appearance synthesizer model $S_\theta$, which takes the same text tokens and the visual tokens of the pose video, noised where the token contains skeletons in the "targeted noise injection" block, and denoises RGB video tokens. At inference time, both models are used sequentially, with the generated pose video of the first model being used to control the generation of the second one.
• Figure 3: Comparison of intermediate representations $\mathcal{M}$. HDMaps are abstract and non-scalable. Panoptic segmentation is 3D-blind and struggles with pedestrian details. Our pose-based representation provides a scalable, 3D-aware, and object-centric structure that is ideal for both forecasting and synthesis.
• Figure 4: Illustration of our visual ego-motion representation, showing three frames from a car turning left. The ego-camera is surrounded by a static, textured sphere and dust particles. Rotations can be inferred from the background's apparent motion (moving right), while speed is encoded by the parallax motion of the particles. Blue dots represent the car's trajectory, and the markings show how the movement between sphere texture and static elements in the background are aligned.
• Figure 5: Human preference study comparing MAD-LTX against competing models. Bars show how often each model is preferred in head-to-head comparisons. '*' denotes proprietary models which use private datasets and significant computational resources.