ConsisDrive: Identity-Preserving Driving World Models for Video Generation by Instance Mask

TL;DR

ConsisDrive tackles identity drift in driving world models by enforcing instance-level temporal consistency through Instance-Masked Attention and Instance-Masked Loss. It adds per-instance identity and trajectory masks to a diffusion-transformer video generator and emphasizes foreground supervision via probabilistic masking. On nuScenes, it achieves state-of-the-art video realism (FVD=37.23, FID=3.88) and strong downstream performance in perception and tracking when used for data augmentation, demonstrating robust instance attribute binding. The approach is compatible with existing foundation models and control signals, enabling scalable, identity-preserving driving video generation with practical impact for perception and planning tasks.

Abstract

Autonomous driving relies on robust models trained on large-scale, high-quality multi-view driving videos. Although world models provide a cost-effective solution for generating realistic driving data, they often suffer from identity drift, where the same object changes its appearance or category across frames due to the absence of instance-level temporal constraints. We introduce ConsisDrive, an identity-preserving driving world model designed to enforce temporal consistency at the instance level. Our framework incorporates two key components: (1) Instance-Masked Attention, which applies instance identity masks and trajectory masks within attention blocks to ensure that visual tokens interact only with their corresponding instance features across spatial and temporal dimensions, thereby preserving object identity consistency; and (2) Instance-Masked Loss, which adaptively emphasizes foreground regions with probabilistic instance masking, reducing background noise while maintaining overall scene fidelity. By integrating these mechanisms, ConsisDrive achieves state-of-the-art driving video generation quality and demonstrates significant improvements in downstream autonomous driving tasks on the nuScenes dataset. Our project page is https://shanpoyang654.github.io/ConsisDrive/page.html.

Figures (5)

• Figure 1: Limitations of Prior Works in Instance Identity Preservation Across Frames.(a)Category Shift: In DriveDreamer2 drivedreamer2, the bus gradually turns into a truck, indicating a failure to preserve semantic identity over time. (b)Color Shift: In MagicDrive-V2 gao2024magicdrivedithighresolutionlongvideo, the car’s color changes inconsistently across frames, violating temporal appearance consistency. (c)Foreground Dilution: In Panacea wen2024panacea, scene-level supervision dilutes supervision over critical foreground regions, breaking temporal identity consistency for small instances like pedestrians. In contrast, our method explicitly enforces instance-level temporal constraints, maintaining consistency across frames and effectively addressing these issues.
• Figure 2: Overview. (a) Instance-Masked Attention, which explicitly directs the model’s attention to each individual instance by incorporating both an instance identity mask and trajectory mask. (b) Instance-Masked Loss Supervision, a probabilistic instance-focused training objective that employs instance loss masks to emphasize supervision on foreground regions. (c) Instance Mask Construction. Illustration of how the Instance Identity Mask, Instance Trajectory Mask, and Instance Loss Mask are constructed from 3D box projections.
• Figure 3: Ablation study of the three key modules. (a) Removing the Identity Mask leads to incorrect instance category rendering, e.g., a traffic cone turns into a crouching pedestrian. (b) Removing the Trajectory Mask results in color shifts of the car. (c) Removing foreground supervision causes blurred generation of small objects, such as pedestrians.
• Figure 4: Backbone Model Generalization to Wan 2.1. (a) The baseline Wan 2.1 + ControlNet (without ConsisDrive) still exhibits significant identity shift in the white car's shape. (b) After plugging in our Instance-Masked Attention and Instance-Masked Loss modules, the system achieves noticeably improved instance consistency across frames.
• Figure 5: Challenges for Non-Continuous Trajectory. (a) Without the Trajectory Mask in the Instance-Masked Attention module, the car is occluded and reappears, but its color changes from white to black. (b) With our Instance-Masked Attention, the car preserves its identity and features even after occlusion and reappearance.