Atlanta Scaled layouts from non-central panoramas

TL;DR

This work tackles scaled 3D indoor layout recovery from a single non-central panorama by integrating a deep network with a geometry-driven pipeline that exploits toroidal non-central projection. A modified HorizonNet (Non-central HorizonNet) extracts structural-line boundaries and corners, followed by two solvers that handle Manhattan and Atlanta world layouts to recover metric room layouts and direct scale estimation without additional measurements. The approach outperforms state-of-the-art methods in both line extraction and layout reconstruction on synthetic data, and demonstrates strong results on real panoramas while handling occlusions. The contributions include two novel geometric solvers, the first deep learning application to non-central panoramas, a synthetic non-central panorama dataset, and the first demonstration of scaled layout recovery from a single panorama.

Abstract

In this work we present a novel approach for 3D layout recovery of indoor environments using a non-central acquisition system. From a non-central panorama, full and scaled 3D lines can be independently recovered by geometry reasoning without geometric nor scale assumptions. However, their sensitivity to noise and complex geometric modeling has led these panoramas being little investigated. Our new pipeline aims to extract the boundaries of the structural lines of an indoor environment with a neural network and exploit the properties of non-central projection systems in a new geometrical processing to recover an scaled 3D layout. The results of our experiments show that we improve state-of-the-art methods for layout reconstruction and line extraction in non-central projection systems. We completely solve the problem in Manhattan and Atlanta environments, handling occlusions and retrieving the metric scale of the room without extra measurements. As far as the authors knowledge goes, our approach is the first work using deep learning on non-central panoramas and recovering scaled layouts from single panoramas.

Figures (11)

• Figure 1: Central (top-left) and non-central (bottom-left) panoramas from the same virtual environment taken in the same position. Both panoramas have similar appearance but there are subtle differences in favor of the second if we want to obtain 3D information. On the right, the scaled layout from a single non-central panorama in Atlanta world. The green wireframe shows the real 3D layout of the virtual environment.
• Figure 2: Toroidal projection of non-central circular panoramas. For each point of the circular trajectory of optical centers with radius $R_c$, there is a ring in which the projection is central. A 3D line $\mathbf{L}$ is projected in the toroidal surface. A projecting ray $\boldsymbol{\Xi}$ intersects the line giving a point in the toroidal surface for a unique optical center.
• Figure 3: Pipeline of our proposal. In a first stage, the neural network extracts the boundaries of the structural lines of the room as well as a probability of corner positions from the non-central panorama. On a second stage, our proposed geometrical processing exploits the properties of non-central projection systems to recover the 3D of the layout from the information provided by the network.
• Figure 4: The non-central circular panorama is processed by Non-central HorizonNet, which is an adaptation of the work sun2019horizonnet. First, it goes through a ResNet50, where high and low-level features are extracted. After a set of convolutions, the result is concatenated and fed to an array of bidirectional LSTMs. The network provides the boundaries of the structural lines of ceiling and floor, as well as the corners of the room as three separate 1D arrays.
• Figure 5: Rays and wall parameter definition. The wall reference system is defined as $\{\mathbf{e}_1,\mathbf{e}_2,\mathbf{e}_3\}$; $\boldsymbol{\Xi}$ and $\boldsymbol{X}$ are the projecting rays; $\mathbf{(l,\bar{l})}$ and $\mathbf{(m,\bar{m})}$ are the ceiling and floor lines that define the wall; $\mathbf{x_L,x_M}$ are the closest points of the lines to the reference system; $h_c$, $h_f$ and $d$ are the distance from the referende system to the ceiling, floor and wall planes respectively.