Unposed Sparse Views Room Layout Reconstruction in the Age of Pretrain Model

TL;DR

This work tackles the challenge of estimating 3D indoor room layouts from unposed, sparse-view RGB images. It introduces Plane-DUSt3R, a plane-focused adaptation of the 3D foundation model DUSt3R, to predict structural plane pointmaps and infer layouts without explicit camera poses, extended to a multi-view setting. The pipeline comprises a 2D plane detector, Plane-DUSt3R for 3D plane information and cross-view correspondence, and a post-processing stage that merges per-view results via alignment and a minimum-cut merge, with a metric-scale variant for scale accuracy. Experiments on Structure3D show state-of-the-art performance in multi-view layout estimation, with strong generalization to in-the-wild and out-of-domain data, and the method demonstrates robust cross-view plane reconstruction for practical 3D room understanding tasks.

Abstract

Room layout estimation from multiple-perspective images is poorly investigated due to the complexities that emerge from multi-view geometry, which requires muti-step solutions such as camera intrinsic and extrinsic estimation, image matching, and triangulation. However, in 3D reconstruction, the advancement of recent 3D foundation models such as DUSt3R has shifted the paradigm from the traditional multi-step structure-from-motion process to an end-to-end single-step approach. To this end, we introduce Plane-DUSt3R, a novel method for multi-view room layout estimation leveraging the 3D foundation model DUSt3R. Plane-DUSt3R incorporates the DUSt3R framework and fine-tunes on a room layout dataset (Structure3D) with a modified objective to estimate structural planes. By generating uniform and parsimonious results, Plane-DUSt3R enables room layout estimation with only a single post-processing step and 2D detection results. Unlike previous methods that rely on single-perspective or panorama image, Plane-DUSt3R extends the setting to handle multiple-perspective images. Moreover, it offers a streamlined, end-to-end solution that simplifies the process and reduces error accumulation. Experimental results demonstrate that Plane-DUSt3R not only outperforms state-of-the-art methods on the synthetic dataset but also proves robust and effective on in the wild data with different image styles such as cartoon. Our code is available at: https://github.com/justacar/Plane-DUSt3R

Figures (14)

• Figure 1: We present a novel method for estimating room layouts from a set of unconstrained indoor images. Our approach demonstrates robust generalization capabilities, performing well on both in-the-wild datasets zhou2018stereo and out-of-domain cartoon weber2023toon3d data.
• Figure 2: Our multi-view room layout estimation pipeline. It consists of three parts: 1) a 2D plane detector $f_1$, 2) a 3D information prediction and correspondence establishment method Plane-DUSt3R $f_2$, and 3) a post-processing algorithm $f_3$.
• Figure 3: Plane-DUSt3R architecture remains identical to DUSt3R. The transformer decoder and regression head are further fine-tuned on the occlusion-free depth map (see Figure \ref{['fig:depth']}).
• Figure 4: The (a) original DUSt3R depth map and (b) occlusion removed depth map.
• Figure 5: (a) Planes are projected onto the x-z plane as 2D line segments. (b) The scene is rotated so that line segments are approximately horizontal or vertical. (c) Line segments are classified and aligned to be either horizontal or vertical. (d) Merged planes are shown, with segments belonging to the same plane indicated by the same color and index.