PlaneMVS: 3D Plane Reconstruction from Multi-View Stereo

TL;DR

PlaneMVS tackles the problem of 3D plane reconstruction from multi-view inputs by introducing a dual-branch architecture: a semantic plane-detection head and a planar MVS head guided by slanted-plane plane sweeping. The slanted-plane hypothesis space enables end-to-end learning of per-pixel plane parameters and a planar depth map within an MVS framework, while a soft-pooling strategy couples 2D plane masks with 3D geometry to produce coherent plane instances. Learned uncertainties balance multiple losses, yielding robust optimization across the plane-detection and MVS tasks. Empirical results on ScanNet, with generalization to 7-Scenes and TUM-RGBD, show PlaneMVS outperforms state-of-the-art single-view plane methods and several learning-based MVS baselines, and ablations confirm the value of slanted planes, soft-pooling, and uncertainty-based training. The approach promises improved indoor scene understanding and AR/robotics applications by delivering accurate, scale-consistent plane geometries over textureless regions.

Abstract

We present a novel framework named PlaneMVS for 3D plane reconstruction from multiple input views with known camera poses. Most previous learning-based plane reconstruction methods reconstruct 3D planes from single images, which highly rely on single-view regression and suffer from depth scale ambiguity. In contrast, we reconstruct 3D planes with a multi-view-stereo (MVS) pipeline that takes advantage of multi-view geometry. We decouple plane reconstruction into a semantic plane detection branch and a plane MVS branch. The semantic plane detection branch is based on a single-view plane detection framework but with differences. The plane MVS branch adopts a set of slanted plane hypotheses to replace conventional depth hypotheses to perform plane sweeping strategy and finally learns pixel-level plane parameters and its planar depth map. We present how the two branches are learned in a balanced way, and propose a soft-pooling loss to associate the outputs of the two branches and make them benefit from each other. Extensive experiments on various indoor datasets show that PlaneMVS significantly outperforms state-of-the-art (SOTA) single-view plane reconstruction methods on both plane detection and 3D geometry metrics. Our method even outperforms a set of SOTA learning-based MVS methods thanks to the learned plane priors. To the best of our knowledge, this is the first work on 3D plane reconstruction within an end-to-end MVS framework. Source code: https://github.com/oppo-us-research/PlaneMVS.

Figures (13)

• Figure 1: Comparison among: (a) single-view plane reconstruction framework, (b) conventional depth-based MVS framework, (c) the proposed multi-view plane reconstruction framework. Our system employs slanted plane hypotheses for plane-sweeping to build the plane MVS branch, which interacts with the plane detection branch. The two branches can benefit from each other.
• Figure 2: The architecture of our proposed plane MVS head. It consists of a plane regression module to regress initial pixel-level plane parameters and a plane refinement module to refine the initial prediction. The key difference of our work from conventional MVS methods is that we apply slanted plane hypotheses for homography warping. The loss objectives during training are omitted for simplicity.
• Figure 3: Qualitative results of reconstructed depth maps on ScanNet dai2017scannet. "PlaneMVS-pixel" denotes the depth reconstructed from pixel-level plane parameters. "PlaneMVS-final" denotes the final depth from the instance plane parameters after plane soft-pooling. Regions with salient differences between our results and others are highlighted with blue and red boxes. Best viewed on screen with zoom-in.
• Figure 4: Qualitative results of plane detection on ScanNet dai2017scannet. The regions with salient differences between our method and PlaneRCNN are highlighted with red boxes.
• Figure 5: Plane hypothesis distribution of the three axes.