PLANA3R: Zero-shot Metric Planar 3D Reconstruction via Feed-Forward Planar Splatting

TL;DR

PLANA3R tackles metric indoor 3D reconstruction from unposed stereo pairs by predicting a sparse, planar-primitives-based scene representation and the relative pose using Vision Transformers, supervised through differentiable planar rendering with depth and normal maps. It removes the need for plane-level annotations or known camera poses, enabling strong zero-shot generalization across diverse indoor environments. The approach introduces a Hierarchical Primitive Prediction Architecture and selective high-resolution primitives, merged via a 3D plane merge step to produce accurate geometry and meaningful plane segmentation, while maintaining efficiency. Empirical results across ScanNetV2, Matterport3D, NYUv2-Plane, and Replica demonstrate state-of-the-art two-view reconstruction, robust monocular depth estimation, and semantic plane segmentation, with scalable training on large-scale datasets.

Abstract

This paper addresses metric 3D reconstruction of indoor scenes by exploiting their inherent geometric regularities with compact representations. Using planar 3D primitives - a well-suited representation for man-made environments - we introduce PLANA3R, a pose-free framework for metric Planar 3D Reconstruction from unposed two-view images. Our approach employs Vision Transformers to extract a set of sparse planar primitives, estimate relative camera poses, and supervise geometry learning via planar splatting, where gradients are propagated through high-resolution rendered depth and normal maps of primitives. Unlike prior feedforward methods that require 3D plane annotations during training, PLANA3R learns planar 3D structures without explicit plane supervision, enabling scalable training on large-scale stereo datasets using only depth and normal annotations. We validate PLANA3R on multiple indoor-scene datasets with metric supervision and demonstrate strong generalization to out-of-domain indoor environments across diverse tasks under metric evaluation protocols, including 3D surface reconstruction, depth estimation, and relative pose estimation. Furthermore, by formulating with planar 3D representation, our method emerges with the ability for accurate plane segmentation. The project page is available at https://lck666666.github.io/plana3r

Figures (12)

• Figure 1: Our proposed Plana3R learns to predict planar 3D primitives and metric-scale relative poses, providing a compact 3D representation of two-view input images with accurate pose estimation, surface geometry, and semantically meaningful 2D & 3D planar segmentation—all in one pass. As a side output, our model also performs well on single-view planar 3D reconstruction.
• Figure 2: Overview of our Plana3R. Given two images captured from the same scene, Plana3R outputs a set of 3D planar primitives and 6-DoF relative camera pose $P_{\text{rel}}$ in metric scale. Plana3R does not perform per-pixel primitive prediction. Instead, it employs a deconvolution network to predict primitives at two distinct resolutions, based on the patch divisions from the ViT encoder. During both training and inference phases, we selectively integrate low-resolution and high-resolution primitives, resulting in a highly compact and expressive representation. Note that even for high-resolution primitives, their total number is only $\frac{H}{8} \times \frac{W}{8}$, which remains highly sparse.
• Figure 3: Comparisons of two-view 3D planar reconstruction on the ScanNetV2 scannet-DaiCSHFN17 (the first row) and the Matterport3D chang2017matterport3d (the last two rows) datasets.
• Figure 4: Comparisons of single-view plane segmentaion and 3D reconstruction on the Replica replica19arxiv.
• Figure 5: Comparisons of two-view 3D plane segmentation on the Matterport3D chang2017matterport3d (the first two rows) and the ScanNetV2 scannet-DaiCSHFN17 (the last row) datasets.