MVSMamba: Multi-View Stereo with State Space Model

TL;DR

MVSMamba tackles the efficiency challenge of learning-based MVS by adopting a Mamba-based architecture that supports linear-time long-range feature modeling. The key innovation is the Dynamic Mamba module (DM-module) with a reference-centered dynamic scanning strategy, enabling omnidirectional, multi-view feature aggregation with minimal overhead. Empirical results on DTU and Tanks-and-Temples show state-of-the-art accuracy and favorable efficiency, supported by extensive ablations that highlight the DM-module’s central role and the effectiveness of the proposed scan strategies. The approach offers practical impact for robust, scalable 3D reconstruction in multi-view setups and establishes a path for further multi-scale Mamba-based interactions in MVS.

Abstract

Robust feature representations are essential for learning-based Multi-View Stereo (MVS), which relies on accurate feature matching. Recent MVS methods leverage Transformers to capture long-range dependencies based on local features extracted by conventional feature pyramid networks. However, the quadratic complexity of Transformer-based MVS methods poses challenges to balance performance and efficiency. Motivated by the global modeling capability and linear complexity of the Mamba architecture, we propose MVSMamba, the first Mamba-based MVS network. MVSMamba enables efficient global feature aggregation with minimal computational overhead. To fully exploit Mamba's potential in MVS, we propose a Dynamic Mamba module (DM-module) based on a novel reference-centered dynamic scanning strategy, which enables: (1) Efficient intra- and inter-view feature interaction from the reference to source views, (2) Omnidirectional multi-view feature representations, and (3) Multi-scale global feature aggregation. Extensive experimental results demonstrate MVSMamba outperforms state-of-the-art MVS methods on the DTU dataset and the Tanks-and-Temples benchmark with both superior performance and efficiency. The source code is available at https://github.com/JianfeiJ/MVSMamba.

Figures (8)

• Figure 1: Comparison of performance vs. efficiency among state-of-the-art CNN-based ($\blacksquare$), Transformer-based ($\CIRCLE$), and our Mamba-based (★) methods on the (a) DTU dataset, the Tanks-and-Temples (b) intermediate and (c) advanced benchmark. The GPU memory and runtime are evaluated on 5-view images with a resolution of 832$\times$1152. The proposed MVSMamba achieves the best performance with superior efficiency.
• Figure 2: Overall architecture of MVSMamba. The proposed Dynamic Mamba module (DM-module) is integrated into the FPN (Sec. \ref{['sec:DM-module']}). First, a reference-centered dynamic scanning strategy extracts four directional feature sequences, which are processed by four independent Mamba blocks. The resulting sequences are then merged back into 2D feature maps. Multi-scale feature aggregation (Sec. \ref{['sec:multi-scale']}) is subsequently performed. Finally, we predicted depth in a coarse-to-fine manner (Sec. \ref{['sec:depth_predict']}).
• Figure 3: Overview of our proposed reference-centered dynamic scanning strategy. (a) Scanning directions of each reference-source feature pairs. (b) Receptive Filed of the reference feature to different source features.
• Figure 4: Qualitative comparison of depth maps in challenging scenarios on the DTU evaluation dataset. Our method predicted more accurate depth maps in texture-less and reflection regions.
• Figure 5: Qualitative comparison of reconstructed point clouds on the Tanks-and-Temples benchmark. The top row shows the precision of Francis ($\tau=5mm$) from the intermediate set, while the bottom row presents the precision of Ballroom ($\tau=10mm$) from the advanced set. Brighter regions indicate lower reconstruction errors under the corresponding distance threshold $\tau$.