CHOSEN: Contrastive Hypothesis Selection for Multi-View Depth Refinement

TL;DR

CHOSEN addresses robustness in depth refinement for multi-view stereo by redefining the solution space into a pseudo disparity domain tied to the capture setup. It employs iterative, PatchMatch-inspired hypothesis sampling (both initial and spatial propagation) and a contrastive, ranking-based selection mechanism to choose the best depth hypotheses, eschewing a fixed probability volume. A lightweight, end-to-end baseline MVS pipeline is augmented with a ranking MLP and context/hypothesis features, trained with a contrastive loss to improve depth and normals while maintaining generalization across datasets and camera configurations. The approach achieves superior depth and normal accuracy on DTU and demonstrates strong generalization to other benchmarks, while remaining relatively parameter-efficient and compatible with existing MVS frameworks. Practical impact includes easier adaptation to diverse capture setups and improved 3D reconstructions in textureless or repetitive regions, with potential for further gains via cost-volume filtering extensions and integration with 3D refinement stages.

Abstract

We propose CHOSEN, a simple yet flexible, robust and effective multi-view depth refinement framework. It can be employed in any existing multi-view stereo pipeline, with straightforward generalization capability for different multi-view capture systems such as camera relative positioning and lenses. Given an initial depth estimation, CHOSEN iteratively re-samples and selects the best hypotheses, and automatically adapts to different metric or intrinsic scales determined by the capture system. The key to our approach is the application of contrastive learning in an appropriate solution space and a carefully designed hypothesis feature, based on which positive and negative hypotheses can be effectively distinguished. Integrated in a simple baseline multi-view stereo pipeline, CHOSEN delivers impressive quality in terms of depth and normal accuracy compared to many current deep learning based multi-view stereo pipelines.

Figures (8)

• Figure 1: CHOSEN's hypotheses sampling and ranking mechanism. Assuming an initial depth estimation, we gather depth hypotheses from its perturbations or its spatial neighbors. For each hypothesis, we combine the matching cost, a second order smoothness term and a context feature to form the input of a learned score function represented by an MLP. The one with the highest score will be selected to update refined depth. The process is performed iteratively.
• Figure 2: Overview of the baseline MVS with our CHOSEN depth refinement. The winner hypothesis is from the initial full range cost volume, followed by applying the hypotheses sampling and best hypothesis selection. Initial sampling and spatial sampling are applied in an alternating fashion. Spatial sampling is facilitated through first order propagation as defined in Eq.\ref{['eq:first order']}. Key to this pipeline is the design of the hypothesis feature, defined in Sec.\ref{['para: hypothesis feature']}. The refined depth is upsampled to the higher resolution using nearest neighbor, and the same refinement procedure will be applied.
• Figure 3: Evolution of the refined depth in our baseline MVS model. The first row shows the ground truth and input images, and we mark each row with the cost volume pyramid configuration. As one can see, the winner-take-all initialization from the cost volume is usually very noisy, but nevertheless contain some accurate estimations. By iteratively re-sample and selecting the best hypotheses, the depth and quality are significantly improved.
• Figure 4: Normal quality comparisons on DTU. Our simple baseline MVS trained only on DTU produces significantly more accurate normals.
• Figure 5: Direct application of our DTU + BlendedMVS trained baseline model on instances from MultiFace knapitsch2017tanks, Tanks & Temples and ETH3D schops2017multi datasets. Our simple baseline achieves consistent generalization ability even though trained on substantially different data.