You Need a Transition Plane: Bridging Continuous Panoramic 3D Reconstruction with Perspective Gaussian Splatting

TL;DR

This work introduces Transition Plane Gaussian Splatting (TPGS) to resolve distortions and boundary discontinuities when reconstructing 3D scenes from panoramic imagery using perspective Gaussian Splatting. By inserting a Transition Plane between cube faces, employing a two-stage intra-to-inter face optimization, and applying spherical sampling for seamless cube-face stitching, the method mitigates cross-face ambiguities and preserves detail across the stitched panorama. The approach builds on 3D Gaussian Splatting within a cubemap/Perspective framework, enabling efficient, differentiable reconstruction that leverages latest perspective-based advances while handling omnidirectional inputs. Extensive experiments on diverse indoor/outdoor and egocentric/roaming datasets demonstrate state-of-the-art performance in PSNR, SSIM, and LPIPS, with practical benefits for VR/AR and autonomous-navigation applications.

Abstract

Recently, reconstructing scenes from a single panoramic image using advanced 3D Gaussian Splatting (3DGS) techniques has attracted growing interest. Panoramic images offer a 360$\times$ 180 field of view (FoV), capturing the entire scene in a single shot. However, panoramic images introduce severe distortion, making it challenging to render 3D Gaussians into 2D distorted equirectangular space directly. Converting equirectangular images to cubemap projections partially alleviates this problem but introduces new challenges, such as projection distortion and discontinuities across cube-face boundaries. To address these limitations, we present a novel framework, named TPGS, to bridge continuous panoramic 3D scene reconstruction with perspective Gaussian splatting. Firstly, we introduce a Transition Plane between adjacent cube faces to enable smoother transitions in splatting directions and mitigate optimization ambiguity in the boundary region. Moreover, an intra-to-inter face optimization strategy is proposed to enhance local details and restore visual consistency across cube-face boundaries. Specifically, we optimize 3D Gaussians within individual cube faces and then fine-tune them in the stitched panoramic space. Additionally, we introduce a spherical sampling technique to eliminate visible stitching seams. Extensive experiments on indoor and outdoor, egocentric, and roaming benchmark datasets demonstrate that our approach outperforms existing state-of-the-art methods. Code and models will be available at https://github.com/zhijieshen-bjtu/TPGS.

Figures (7)

• Figure 1: Comparison of the commonly used framework (a) against our proposed method (b). Our approach introduces a transition plane along with an intra-to-inter face optimization strategy to address the challenges inherent in cubemap projection.
• Figure 2: Panoramic distortion v.s. projection distortion. (a) The spherical image is projected onto a 2D equirectangular panorama. Non-uniform sampling density in the latitude direction causes panoramic distortion. (b) The spherical image is mapped onto six cube faces. Non-uniform sampling density from the center to the edges causes projection distortion, reducing resolution in boundary regions. This uneven resolution disrupts the uniformity of supervision signals during optimization.
• Figure 3: Splatting ambiguity removal via the proposed transition plane. When 3D Gaussians projected onto cube boundaries may span adjacent planes, causing independent updates to the same Gaussian attributes and leading to optimization conflicts and ambiguities. The transition plane method resolves this by splatting Gaussians onto an intermediate plane
• Figure 4: The pipeline of the proposed TPGS.
• Figure 5: Cube padding with spherical sampling.