360-GS: Layout-guided Panoramic Gaussian Splatting For Indoor Roaming

TL;DR

This work addresses the challenge of generating high-quality novel views from sparse 360° indoor panoramas by extending 3D Gaussian Splatting (3D-GS) to panoramic inputs. It introduces 360° Gaussian splatting, which projects 3D Gaussians onto the tangent plane of the unit sphere before mapping to the sphere, enabling direct rendering in equirectangular space. The method leverages room layout priors for both initialization and regularization, producing plausible, artifact-free geometry and reducing floaters in novel views. Experiments on Matterport3D scenes show state-of-the-art performance with real-time panoramic rendering, highlighting the practical impact for immersive indoor roaming with limited inputs.

Abstract

3D Gaussian Splatting (3D-GS) has recently attracted great attention with real-time and photo-realistic renderings. This technique typically takes perspective images as input and optimizes a set of 3D elliptical Gaussians by splatting them onto the image planes, resulting in 2D Gaussians. However, applying 3D-GS to panoramic inputs presents challenges in effectively modeling the projection onto the spherical surface of ${360^\circ}$ images using 2D Gaussians. In practical applications, input panoramas are often sparse, leading to unreliable initialization of 3D Gaussians and subsequent degradation of 3D-GS quality. In addition, due to the under-constrained geometry of texture-less planes (e.g., walls and floors), 3D-GS struggles to model these flat regions with elliptical Gaussians, resulting in significant floaters in novel views. To address these issues, we propose 360-GS, a novel $360^{\circ}$ Gaussian splatting for a limited set of panoramic inputs. Instead of splatting 3D Gaussians directly onto the spherical surface, 360-GS projects them onto the tangent plane of the unit sphere and then maps them to the spherical projections. This adaptation enables the representation of the projection using Gaussians. We guide the optimization of 360-GS by exploiting layout priors within panoramas, which are simple to obtain and contain strong structural information about the indoor scene. Our experimental results demonstrate that 360-GS allows panoramic rendering and outperforms state-of-the-art methods with fewer artifacts in novel view synthesis, thus providing immersive roaming in indoor scenarios.

Figures (9)

• Figure 1: Overview of 360-GS architecture. Given a limited set of panoramas, we estimate the room layout and depth to guide the optimization of 3D Gaussian. These priors are transformed into 3D representations and jointly merged to a point cloud, which is used to initialize 3D Gaussians. We propose a 3D Gaussian splatting algorithm to project 3D Gaussians to panoramic space. Based on the projected Gaussians, we can render panoramas through a differentiable tile rasterizer. To reduce floaters in novel views, we regularize the optimization of 3D Gaussians by minimizing the cosine distance between the movement of position vectors and normals of layout point clouds.
• Figure 2: Illustration of panoramic Gaussian splatting. We show a toy case for splatting a 3D sphere (a special case of 3D Gaussian) onto panoramic images. The 3D sphere is defined by its position $\bm{\mu} \in \mathbb{R}^3$ and its radius $R \in \mathbb{R}$. In the middle column, we visualize the projection of 3D spheres under different positions. In the right column, 3D spheres with varying radii are splatted onto the panoramas. These projections are not elliptical and cannot be accurately modeled with 2D Gaussians, as attempted by 3D-GS.
• Figure 3: Two naive pipelines for applying 3D-GS to panoramic inputs. Left: we split panoramas into perspective images with poses and feed them to 3D-GS. To render panoramas, we render perspective images centered at the camera of panoramas. Subsequently, these images are transformed and stitched into equirectangular projection. These operations introduce stitching artifacts. Right: we fail to directly train 3D-GS with panoramic inputs.
• Figure 4:
• Figure 5: Impact of layout-guided regularization. We present a 2D toy case for optimizing 3D Gaussians, marked with an orange color. Without our regularization, 3D Gaussians gravitate towards the gradient direction, disrupting the inherent layout structure. This results in some Gaussians appearing outside the walls, causing distorted planes and "floaters" in novel views. Our layout-guided regularization effectively preserves the overall structure during optimization.