MP-SfM: Monocular Surface Priors for Robust Structure-from-Motion

TL;DR

MP-SfM addresses the fragility of Structure-from-Motion under extreme viewpoint changes by embedding monocular depth and surface normal priors with uncertainty into an incremental SfM framework. It initializes from two views, registers new views with lifted single-view depth, and alternates single-view priors with multi-view optimization, augmented by a depth-consistency check to reject incorrect registrations. The approach yields significant gains in accuracy and reconstruction completeness in low-overlap and low-parallax scenarios, while preserving performance in standard conditions, and is robust to errors in priors due to principled uncertainty propagation. Practically, MP-SfM lowers the barrier for non-experts to obtain reliable 3D reconstructions in indoor environments and large, textureless scenes, with publicly available code to foster uptake and further improvement as monocular depth models evolve.

Abstract

While Structure-from-Motion (SfM) has seen much progress over the years, state-of-the-art systems are prone to failure when facing extreme viewpoint changes in low-overlap, low-parallax or high-symmetry scenarios. Because capturing images that avoid these pitfalls is challenging, this severely limits the wider use of SfM, especially by non-expert users. We overcome these limitations by augmenting the classical SfM paradigm with monocular depth and normal priors inferred by deep neural networks. Thanks to a tight integration of monocular and multi-view constraints, our approach significantly outperforms existing ones under extreme viewpoint changes, while maintaining strong performance in standard conditions. We also show that monocular priors can help reject faulty associations due to symmetries, which is a long-standing problem for SfM. This makes our approach the first capable of reliably reconstructing challenging indoor environments from few images. Through principled uncertainty propagation, it is robust to errors in the priors, can handle priors inferred by different models with little tuning, and will thus easily benefit from future progress in monocular depth and normal estimation. Our code is publicly available at https://github.com/cvg/mpsfm.

Figures (12)

• Figure 1: A typical failure case for SfM. Existing approaches cannot handle low-overlap image pairs because they require three-view tracks to ensure a consistent scale across the scene. We bridge this limitation by augmenting SfM with monocular depth and normal priors from off-the-shelf deep networks. This makes SfM significantly more robust for data captured by non-expert users.
• Figure 2: Overview of our approach. Given image correspondences, depth, and surface normals, we first initialize the reconstruction by estimating a relative pose or, if the parallax is low, an absolute pose from points lifted to 3D by depth. While SfM can generally estimate 3D only for points observed in multiple views, we leverage single-view observations with depth. This helps registering images with lower visual overlap. Camera poses, 3D points, and depth maps are refined by alternating between bundle adjustment and normal integration with depth constraints. Finally, we reject incorrect registrations, e.g., due to symmetries, by checking that the depth is consistent across views.
• Figure 3: Qualitative results for low overlap scenes. Left: Input images with low overlap. Center: Estimated (red) and ground-truth (blue) camera poses with the monocular depth refined by our system. Right: Lifted refined depth of which points are colored differently whether they are visible in a single image (red), two (green), or at least three images.
• Figure 4: Depth consistency check. These two image pairs are incorrectly matched because of symmetries ( blue points). Our approach successfully rejects them as a large ratio of pixels have an inconsistent depth (red), while ignoring occlusion (yellow) and areas with consistent depth (cyan).
• Figure 5: Qualitative comparison of reconstructions for low-overlap scenes.Estimated (red) and ground-truth (blue) camera poses, and AUC accuracies at $1^\circ/5^\circ/20^\circ$ error thresholds are presented. Left: COLMAP schoenberger2016sfm. Center: MASt3R-SfM duisterhof2024mast3r. Right: Our method. Rows 1–2 show scenes from SMERF duckworth2023smerf, while rows 3–4 are from ETH3D schops2017multi and Tanks and Temples knapitsch2017tanks.