MuM: Multi-View Masked Image Modeling for 3D Vision

TL;DR

MuM introduces a multi-view masked image modeling objective that extends MAE to an arbitrary number of views of a scene, enabling self-supervised learning of geometric 3D representations. The method employs a ViT-L encoder and a multi-view ViT-B decoder with alternating frame-wise and global attention, trained on a diverse mix of ~20 million frames, and uses a uniform masking strategy across views. Empirically, MuM outperforms state-of-the-art 3D vision encoders such as DINOv3 and CroCo v2 on downstream tasks including feedforward reconstruction, dense matching, and relative pose estimation, while using substantially less compute. Although MuM remains slightly behind in some single-view semantic tasks, its strong multi-view performance and data efficiency demonstrate a scalable path for geometry-focused SSL in 3D vision.

Abstract

Self-supervised learning on images seeks to extract meaningful visual representations from unlabeled data. When scaled to large datasets, this paradigm has achieved state-of-the-art performance and the resulting trained models such as DINOv3 have seen widespread adoption. However, most prior efforts are optimized for semantic understanding rather than geometric reasoning. One important exception is Cross-View Completion, CroCo, which is a form of masked autoencoding (MAE) tailored for 3D understanding. In this work, we continue on the path proposed by CroCo and focus on learning features tailored for 3D vision. In a nutshell, we extend MAE to arbitrarily many views of the same scene. By uniformly masking all views and employing a lightweight decoder with inter-frame attention, our approach is inherently simpler and more scalable than CroCo. We evaluate the resulting model, MuM, extensively on downstream tasks including feedforward reconstruction, dense image matching and relative pose estimation, finding that it outperforms the state-of-the-art visual encoders DINOv3 and CroCo v2.

Figures (7)

• Figure 1: MuM SSL pretraining. An arbitrary number of input images from the same scene are first uniformly masked and processed independently in a ViT-L encoder. The representations are then jointly processed in a multi-view ViT-B decoder. The final representations are linearly mapped to pixel space and the reconstruction loss is computed according to Equation \ref{['eq:loss']}.
• Figure 2: Training dynamics. Dense matching with a linear probe and semantic segmentation performance during 500K steps of training.
• Figure 3: Data example. We illustrate a sequence sampled from the data and the predicted reconstructions. Note, as we train with a normalized pixel objective, we visualize the prediction plus the patch mean. Our data samples generally feature difficult viewpoint changes and varying co-visibility. Details about our sampling technique are provided in Appendix \ref{['appendix:pretrain']}.
• Figure 4: Feature layer. We report EPE for linear probing on MegaDepth. The later layers provide better representations.
• Figure 5: Data examples. We visualize random sampled sequences from the dataset. Sequence lengths vary.