MIM4D: Masked Modeling with Multi-View Video for Autonomous Driving Representation Learning

TL;DR

MIM4D presents a dual masked image modeling framework that extends MAE-style pretraining into 4D by jointly modeling spatial multi-view structure and temporal dynamics from masked multi-view video. It builds 3D voxel features via a voxel encoder, reconstructs dropped voxel frames through a temporal BEV-based decoder, and supervises geometry with differentiable volume rendering using a continuous scene flow representation and an SDF-based rendering pipeline. The approach yields state-of-the-art or strong improvements on nuScenes across BEV segmentation, 3D object detection, and HD map construction, without relying on dense 3D annotations. This work demonstrates the effectiveness of 4D self-supervised pretraining for scalable autonomous driving representations and highlights the practical impact on perception tasks in dynamic environments.

Abstract

Learning robust and scalable visual representations from massive multi-view video data remains a challenge in computer vision and autonomous driving. Existing pre-training methods either rely on expensive supervised learning with 3D annotations, limiting the scalability, or focus on single-frame or monocular inputs, neglecting the temporal information. We propose MIM4D, a novel pre-training paradigm based on dual masked image modeling (MIM). MIM4D leverages both spatial and temporal relations by training on masked multi-view video inputs. It constructs pseudo-3D features using continuous scene flow and projects them onto 2D plane for supervision. To address the lack of dense 3D supervision, MIM4D reconstruct pixels by employing 3D volumetric differentiable rendering to learn geometric representations. We demonstrate that MIM4D achieves state-of-the-art performance on the nuScenes dataset for visual representation learning in autonomous driving. It significantly improves existing methods on multiple downstream tasks, including BEV segmentation (8.7% IoU), 3D object detection (3.5% mAP), and HD map construction (1.4% mAP). Our work offers a new choice for learning representation at scale in autonomous driving. Code and models are released at https://github.com/hustvl/MIM4D

Figures (2)

• Figure 1: The overall architecture of MIM4D. We take the masked multi-frame multi-view images as input and utilize a voxel encoder to obtain the voxel feature sequence. Specifically, we utilize an image backbone network to extract hierarchical features, following which a view transformation module is employed to convert the multi-view features into representations in 3D space. After that, we drop the voxel feature of the m-th frame and reconstruct it base on the remaining voxel feature sequence using the voxel decoder module. Finally, the reconstructed $\hat{\textbf{V}}_{t-m}$ is fed into the volume-based neural rendering decoder, where it is projected back onto a 2D plane. The 2D pixel-wise RGB and depth are used as supervision.
• Figure 2: The architecture of voxel decoder. Taking the voxel sequence as input, it is first processed by the Height2Channel module to convert voxel features into BEV features, temporarily compressing the height dimension. Next, we use the long-term temporal transformer and the short-term temporal transformer to extract information from the remaining BEV feature sequence, which are then fused together to reconstruct the dropped BEV feature. The symbol © denotes feature concatenation and fusion. Finally, we employ the Channel2Height module to convert the reconstructed BEV feature back to voxel feature.