OmniGS: Fast Radiance Field Reconstruction using Omnidirectional Gaussian Splatting

TL;DR

OmniGS, a novel omnidirectional Gaussian splatting system, to take advantage of omnidirectional images for fast radiance field reconstruction, and realizes differentiable optimization of the omnidirectional radiance field without the requirement of cube-map rectification or tangent-plane approximation.

Abstract

Photorealistic reconstruction relying on 3D Gaussian Splatting has shown promising potential in various domains. However, the current 3D Gaussian Splatting system only supports radiance field reconstruction using undistorted perspective images. In this paper, we present OmniGS, a novel omnidirectional Gaussian splatting system, to take advantage of omnidirectional images for fast radiance field reconstruction. Specifically, we conduct a theoretical analysis of spherical camera model derivatives in 3D Gaussian Splatting. According to the derivatives, we then implement a new GPU-accelerated omnidirectional rasterizer that directly splats 3D Gaussians onto the equirectangular screen space for omnidirectional image rendering. We realize differentiable optimization of the omnidirectional radiance field without the requirement of cube-map rectification or tangent-plane approximation. Extensive experiments conducted in egocentric and roaming scenarios demonstrate that our method achieves state-of-the-art reconstruction quality and high rendering speed using omnidirectional images. The code will be publicly available.

Figures (6)

• Figure 1: We introduce OmniGS, a novel omnidirectional radiance field reconstruction method. It takes a series of calibrated monocular 360-degree images and sparse SfM point clouds as input to quickly recover 3D Gaussians as large-scale scene representations, achieving real-time omnidirectional novel view synthesis.
• Figure 2: Coordinate systems used in OmniGS. We use the SLAM convention for cameras, i.e. +X is right, +Y is down, and +Z is forward. In the forward rendering process, 3D Gaussians are first transformed from the world coordinate system to the camera space, then projected onto the image. The latitude-longitude coordinate system and the uniform screen-space coordinate system serve as intermediate variables during the projection process. The X-Z plane of the camera space is the equatorial plane, i.e. $lat=0$.
• Figure 3: A schematic overview of OmniGS optimization flow. It optimized 3D Gaussian representation by minimicing the loss between the rendering omnidirectional images and the input ground truth images.
• Figure 4: Qualitative comparison example of novel-view synthesis on 360Roam dataset. OmniGS can reconstruct clearer detail structures. It is also free from obvious holes or blurs, as the results in cafe show.
• Figure 5: Qualitative comparisons of omnidirectional novel-view synthesis in egocentric scenes. OmniGS can reconstruct the details more sharply and precisely with less training time, i.e. 25 minutes.