Muskie: Multi-view Masked Image Modeling for 3D Vision Pre-training

TL;DR

Muskie introduces a native multi-view vision backbone pre-trained with Multi-view Masked Image Modeling to enforce cross-view consistency and geometric reasoning without 3D supervision. The method uses Alternating Attention and an aggressive masking strategy to reconstruct heavily masked regions across views, enabling implicit discovery of geometric correspondences. Empirical results show superior multi-view correspondence and improved 3D reconstruction and camera pose estimation when Muskie is used as a backbone, across multiple datasets. The work demonstrates that multi-view pre-training can imbue vision backbones with robust 3D-aware representations, enhancing practical 3D tasks without explicit 3D annotations.

Abstract

We present Muskie, a native multi-view vision backbone designed for 3D vision tasks. Unlike existing models, which are frame-wise and exhibit limited multi-view consistency, Muskie is designed to process multiple views simultaneously and introduce multi-view consistency in pre-training stage. Muskie is trained to reconstruct heavily masked content in one view by finding and utilizing geometric correspondences from other views. Through this pretext task and our proposed aggressive masking strategy, the model implicitly to learn view-invariant features and develop strong geometric understanding without any 3D supervision. Compared with state-of-the-art frame-wise backbones such as DINO, Muskie achieves higher multi-view correspondence accuracy. Furthermore, we demonstrate that using Muskie as a backbone consistently enhances performance on downstream 3D tasks, including camera pose estimation and pointmap reconstruction. Codes are publicly available at https://leo-frank.github.io/Muskie/

Figures (14)

• Figure 1: Muskie is a native multi-view visual backbone designed for 3D vision tasks. Through Multi-view Masked Image Modeling pre-training, it learns to jointly extract representations across multiple views in a single forward pass. This is in contrast to the conventional frame-wise paradigm, where a ViT independently encodes each view before features are fused. Muskie establishes stronger multi-view consistency, demonstrated by predicted tracks (red) that align closely with the ground truth (green). Using Muskie as a backbone also leads to superior performance in applications like pointmap estimation, where it produces coherent and geometrically complete 3D reconstructions.
• Figure 2: Overview of Muskie architecture. Multi-view images are divided into patches, and a portion of them is masked using various masking shapes (random, rectangular, or elliptical), replaced with learnable tokens. A subset of views is kept unmasked to serve as reference for others. These patches are jointly processed via stacked alternating-attention blockswang2025vggt. A lightweight linear head reconstructs the masked patches along with confidence maps. For comparison, MAEhe21masked performs Masked Image Modeling (MIM) in a single-view setting, while CroCoweinzaepfel2022crocoweinzaepfel2023crocov2 extends MIM to dual views but still encodes each view independently during the encoding stage.
• Figure 3: Visualizations of the reconstructions from masked images. Samples are taken from ETH3Deth3d and 7Scenes7scenes. Left: The model successfully reconstructs co-visible regions. Meanwhile, for areas that are not co-visible (e.g., newly exposed surfaces due to camera motion), the model produces a blurry reconstruction and assigns a low confidence score. Right: In a more challenging scenario where the reference views provide little information, the model can aggregate information from sparse cues to reconstruct.
• Figure 4: Qualitative comparison of predicted pointmaps for 7Scenes7scenes and NRGBDnrgbd_dataset_cvpr22. We compare the 3D reconstruction results of our method against baselines using different visual backbones. Our method consistently produces reconstructions that more complete and more faithful to the ground truth geometry across all scenes.
• Figure 5: Qualitative comparison of multi-view correspondence estimation. Sequences are sampled from the NAVIjampani2023navi, ScanNetdai2017scannet, and NRGBDnrgbd_dataset_cvpr22. We visualize the predicted tracks (red) and the ground-truth tracks (green). Points are only rendered in frames where the GT point is visible. The close alignment between the red and green paths highlights our method's ability to maintain accurate consistency.