Prism: Semi-Supervised Multi-View Stereo with Monocular Structure Priors

TL;DR

Prism tackles the domain gap in multi-view stereo by enabling semi-supervised training on real unlabeled videos and photorealistic synthetic data. It introduces monocular relative-depth priors learned on synthetic data and two perceptual-inspired losses— a deep feature loss and a pyramid-style SSIM loss—to transfer structured depth information to MVS predictions on real data. The approach yields large gains over unsupervised and synthetic-supervised baselines on ScanNet++ and ARKitScenes, demonstrating robust handling of challenging indoor scenes and reflective or textureless regions. By bridging real-world and synthetic data with monocular priors, Prism enables more scalable training for 3D reconstruction in practical applications.

Abstract

The promise of unsupervised multi-view-stereo (MVS) is to leverage large unlabeled datasets, yet current methods underperform when training on difficult data, such as handheld smartphone videos of indoor scenes. Meanwhile, high-quality synthetic datasets are available but MVS networks trained on these datasets fail to generalize to real-world examples. To bridge this gap, we propose a semi-supervised learning framework that allows us to train on real and rendered images jointly, capturing structural priors from synthetic data while ensuring parity with the real-world domain. Central to our framework is a novel set of losses that leverages powerful existing monocular relative-depth estimators trained on the synthetic dataset, transferring the rich structure of this relative depth to the MVS predictions on unlabeled data. Inspired by perceptual image metrics, we compare the MVS and monocular predictions via a deep feature loss and a multi-scale statistical loss. Our full framework, which we call Prism, achieves large quantitative and qualitative improvements over current unsupervised and synthetic-supervised MVS networks. This is a best-case-scenario result, opening the door to using both unlabeled smartphone videos and photorealistic synthetic datasets for training MVS networks.

Figures (6)

• Figure 1: Using structure priors from a monocular relative-depth network, Prism effectively trains with a combination of real unlabeled smartphone video and synthetic data. It outperforms all 3 baselines: unsupervised on smartphone data (A), supervised on synthetic data (B), and semi-supervised using both (C). Results shown are for the ScanNet++ dataset dai2017scannet. See Sec. \ref{['sec:exp']} for details.
• Figure 2: Overview of Prism framework for semi-supervised MVS. We leverage both real unlabeled smartphone data and labeled synthetic data to train MVS networks. Our central idea is to (1) train an existing monocular relative-depth prediction network on the synthetic set in order to capture high-quality structure priors and (2) teach the MVS network to use these structural priors on the unlabeled set via losses inspired by perceptual image metrics. In addition to these monocular losses, we also utilize unsupervised losses on the unlabeled real examples and supervised losses on the synthetic examples.
• Figure 3: Visual Ablation Study. Each component of Prism interacts constructively, helping bring out fine detail (top row) and global structure (bottom row). The monocular structure prior significantly helps in the case of textureless/reflective surfaces. The supervised loss helps add a final smoothness to flat surfaces and sharpness to object boundaries. See Sec. \ref{['sec:exp_abl']} for details and Table \ref{['tab:ablation']} for quantitative results.
• Figure 4: Qualitative Results. Prism outperforms all baselines, producing sharp and accurate depth maps with excellent global structure and fine-grained local detail. With the arrows, we indicate hard cases where Prism performs well: textureless and reflective surfaces (rows 1, 2, 6) and thin structures (rows 1, 3, 4, 5). In our ablation study, we find the monocular losses largely help with textureless and reflective surfaces, while all components interact constructively to improve thin structures.
• Figure 5: Visual Ablation Study. We isolate the effect of various monocular losses. Only our multi-scale loss $\ell_\mathrm{ssim}$ and our deep feature loss $\ell_\mathrm{feat}$ can handle confusing geometry (top row) and textureless surfaces (bottom row). See Table \ref{['tab:ablation']} for quantitative results.