MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo

TL;DR

MVSFormer++ tackles the challenge of effectively incorporating transformer architectures into multi-view stereo (MVS) by splitting attention strategies across modules and integrating cross-view information into a frozen ViT backbone. It introduces Side View Attention (SVA) to enhance feature encoding and Cost Volume Transformer (CVT) with Frustoconical Positional Encoding (FPE) and Adaptive Attention Scaling (AAS) to stabilize and improve cost-volume regularization, supplemented by normalization strategies. The approach achieves state-of-the-art results on DTU and Tanks-and-Temples, with strong ETH3D performance, and ablations confirm the critical roles of 3D positional encoding, attention scaling, and per-module attention customization. These findings demonstrate that thoughtfully tailored transformer designs can substantially boost dense 3D reconstruction quality and generalization to high-resolution outdoor scenes.

Abstract

Recent advancements in learning-based Multi-View Stereo (MVS) methods have prominently featured transformer-based models with attention mechanisms. However, existing approaches have not thoroughly investigated the profound influence of transformers on different MVS modules, resulting in limited depth estimation capabilities. In this paper, we introduce MVSFormer++, a method that prudently maximizes the inherent characteristics of attention to enhance various components of the MVS pipeline. Formally, our approach involves infusing cross-view information into the pre-trained DINOv2 model to facilitate MVS learning. Furthermore, we employ different attention mechanisms for the feature encoder and cost volume regularization, focusing on feature and spatial aggregations respectively. Additionally, we uncover that some design details would substantially impact the performance of transformer modules in MVS, including normalized 3D positional encoding, adaptive attention scaling, and the position of layer normalization. Comprehensive experiments on DTU, Tanks-and-Temples, BlendedMVS, and ETH3D validate the effectiveness of the proposed method. Notably, MVSFormer++ achieves state-of-the-art performance on the challenging DTU and Tanks-and-Temples benchmarks.

Figures (13)

• Figure 1: (a) Point cloud results compared between MVSFormer cao2022mvsformer and the proposed MVSFormer++ on DTU and Tanks-and-Temples. Results of state-of-the-art MVS methods on (b) DTU and (c) Tanks-and-Temples benchmark.
• Figure 2: The Overview of MVSFormer++. (a) Feature extraction enhanced with SVA module, normalized 2D-PE, and Norm&ALS. (b) Multi-scale cost volume formation and regularization, where CVT is strengthed by FPE and AAS resulting in solid depth estimation.
• Figure 3: Illustration of SVA. Self and cross-view attention are separately used to learn reference and source features respectively.
• Figure 4: Illustration of 3D FPE and attention dilution. (a) We normalize all points in the cost volume within the nearest and farthest depth plane. (b) The attention score would be diluted when the sequence increases, making it challenging to correctly focus on related target values.
• Figure 5: Qualitative results compared with state-of-the-art models on scan77 in DTU.