RIAV-MVS: Recurrent-Indexing an Asymmetric Volume for Multi-View Stereo

TL;DR

A “learning-to-optimize” paradigm that iteratively indexes a plane-sweeping cost volume and regresses the depth map via a convolutional Gated Recurrent Unit (GRU) and incorporates a residual pose network to correct the relative poses is presented.

Abstract

This paper presents a learning-based method for multi-view depth estimation from posed images. Our core idea is a "learning-to-optimize" paradigm that iteratively indexes a plane-sweeping cost volume and regresses the depth map via a convolutional Gated Recurrent Unit (GRU). Since the cost volume plays a paramount role in encoding the multi-view geometry, we aim to improve its construction both at pixel- and frame- levels. At the pixel level, we propose to break the symmetry of the Siamese network (which is typically used in MVS to extract image features) by introducing a transformer block to the reference image (but not to the source images). Such an asymmetric volume allows the network to extract global features from the reference image to predict its depth map. Given potential inaccuracies in the poses between reference and source images, we propose to incorporate a residual pose network to correct the relative poses. This essentially rectifies the cost volume at the frame level. We conduct extensive experiments on real-world MVS datasets and show that our method achieves state-of-the-art performance in terms of both within-dataset evaluation and cross-dataset generalization. Code available: https://github.com/oppo-us-research/riav-mvs.

Figures (11)

• Figure 1: Our pipeline versus RAFT raft-teed2020 and IterMVS itermvs_wang2021. Our recurrent processing of a plane-sweep cost volume by the iteratively refined index field serves as a new design for multi-view depth estimation.
• Figure 2: Architecture of our proposed network. It consists of a feature extraction (i.e., F-Net, a Transformer, and C-Net) block, a cost volume construction and index field GRU-based optimization block, and a residual pose block.
• Figure 3: Qualitative results on ScanNet dai2017scannet (top two rows) and DTU dtu_jensen2014large test set. Left two columns show reference image and ground truth depth, and other columns are the estimated depth by baseline IterMVS itermvs_wang2021, PairNet deepvideomvs2021 and ours (the full version), respectively. Our method outperform the baselines on thin structures, small objects and boundaries, as highlighted in green for ours and in red for the baselines. The abs-err errors (in meters) are imposed on the depth maps for comparison.
• Figure 4: Multi-scale feature fusion layer.
• Figure 5: Training logs at last logging step on ScanNet dai2017scannet training set. Columns show samples and results of mini-batch ones b0, b1, b2, and b3. For the training logs, we show the color maps of the ground truth depths and predictions in the inverse space (i.e., disparity), so as to better align with the training loss calculated on the inverse depth domain.