PanoTPS-Net: Panoramic Room Layout Estimation via Thin Plate Spline Transformation

TL;DR

PanoTPS-Net tackles 3D room layout estimation from a single panorama by predicting Thin Plate Spline (TPS) transformation parameters via a CNN and warping a reference layout to the target. This end-to-end image-warping approach enables robust handling of both cuboid and non-cuboid layouts, achieving competitive or state-of-the-art $3DIoU$ and $2DIoU$ scores on PanoContext, Stanford-2D3D, Matterport3DLayout, and ZInD datasets. The method uses edge and corner maps, a MXception-based feature extractor, and corner-map post-processing for non-cuboid cases, with an efficient runtime (~5 FPS) and a total of 22.3 million parameters. Ablation studies identify optimal configurations (e.g., loss weights, number of TPS control points) and demonstrate the benefits of simultaneous edge+corner warping, while acknowledging occlusion as a limitation and proposing future two-stage strategies to mitigate it. The work also suggests a broader applicability of TPS-based warping to other vision tasks beyond room layout estimation.

Abstract

Accurately estimating the 3D layout of rooms is a crucial task in computer vision, with potential applications in robotics, augmented reality, and interior design. This paper proposes a novel model, PanoTPS-Net, to estimate room layout from a single panorama image. Leveraging a Convolutional Neural Network (CNN) and incorporating a Thin Plate Spline (TPS) spatial transformation, the architecture of PanoTPS-Net is divided into two stages: First, a convolutional neural network extracts the high-level features from the input images, allowing the network to learn the spatial parameters of the TPS transformation. Second, the TPS spatial transformation layer is generated to warp a reference layout to the required layout based on the predicted parameters. This unique combination empowers the model to properly predict room layouts while also generalizing effectively to both cuboid and non-cuboid layouts. Extensive experiments on publicly available datasets and comparisons with state-of-the-art methods demonstrate the effectiveness of the proposed method. The results underscore the model's accuracy in room layout estimation and emphasize the compatibility between the TPS transformation and panorama images. The robustness of the model in handling both cuboid and non-cuboid room layout estimation is evident with a 3DIoU value of 85.49, 86.16, 81.76, and 91.98 on PanoContext, Stanford-2D3D, Matterport3DLayout, and ZInD datasets, respectively. The source code is available at: https://github.com/HatemHosam/PanoTPS_Net.

Figures (8)

• Figure 1: Overview of the proposed method. The yellow points represent the source control point of the TPS transformation in the reference edge map, while the orange points represent the predicted control points in the target edge map.
• Figure 2: The proposed PanoTPS-Net architecture for 3D room layout estimation. (a) A modified Xception architecture (each layer shows the number of output features) is employed to extract the TPS transformation parameters, then those parameters are used by the TPS transformer layer to warp the reference layout (edge and corner maps) to the target layout. (b) The corner map post-processing is applied in the case of non-cuboid room layout estimation.
• Figure 3: The proposed Corner map post-processing stage. This stage is applied in the case of non-cuboid room layout estimation. This stage includes passing the initially predicted corner map to a binarization stage, connected component detection, corner size analysis, and finally splitting corners into multiple corners based on a predefined criteria.
• Figure 4: Sample results obtained by the proposed method. The top four rows represent sample results from PanoContext, and the bottom four rows represent sample results from Stanford-2D3D.
• Figure 5: Sample results obtained by the proposed method on Matterport3DLayout (the top 4 rows) and Zillow indoor dataset (the bottom four rows) for non-cuboid room layout estimation. The results show the final corner/edge maps after corner map post-processing shown in Fig.\ref{['fig2-a']}