SC-OmniGS: Self-Calibrating Omnidirectional Gaussian Splatting

TL;DR

SC-OmniGS tackles the challenge of reconstructing omnidirectional radiance fields from 360-degree images when camera poses and distortions are uncertain. It introduces a differentiable omnidirectional rasterizer, a learnable omnidirectional camera model, and 3D Gaussian splatting to jointly optimize scene representation, poses, and distortion parameters, all guided by a weighted spherical photometric loss. The approach demonstrates robust calibration and high-fidelity reconstructions on synthetic and real datasets, outperforming both non-calibrated baselines and prior self-calibration methods, even when training from scratch or with noisy poses. This work enables fast, accurate, and distortion-aware omnidirectional scene understanding with practical implications for VR, robotics, and SLAM.

Abstract

360-degree cameras streamline data collection for radiance field 3D reconstruction by capturing comprehensive scene data. However, traditional radiance field methods do not address the specific challenges inherent to 360-degree images. We present SC-OmniGS, a novel self-calibrating omnidirectional Gaussian splatting system for fast and accurate omnidirectional radiance field reconstruction using 360-degree images. Rather than converting 360-degree images to cube maps and performing perspective image calibration, we treat 360-degree images as a whole sphere and derive a mathematical framework that enables direct omnidirectional camera pose calibration accompanied by 3D Gaussians optimization. Furthermore, we introduce a differentiable omnidirectional camera model in order to rectify the distortion of real-world data for performance enhancement. Overall, the omnidirectional camera intrinsic model, extrinsic poses, and 3D Gaussians are jointly optimized by minimizing weighted spherical photometric loss. Extensive experiments have demonstrated that our proposed SC-OmniGS is able to recover a high-quality radiance field from noisy camera poses or even no pose prior in challenging scenarios characterized by wide baselines and non-object-centric configurations. The noticeable performance gain in the real-world dataset captured by consumer-grade omnidirectional cameras verifies the effectiveness of our general omnidirectional camera model in reducing the distortion of 360-degree images.

Figures (12)

• Figure 1: SC-OmniGS jointly optimizes the omnidirectional camera model, poses, and 3D Gaussians using a differentiable omnidirectional rasterizer. It can achieve rapid radiance field reconstruction with no pose prior and render high-fidelity novel views.
• Figure 2: A schematic overview of SC-OmniGS optimization flow.
• Figure 3: Differentiable omnidirectional camera model.
• Figure 4: Qualitative comparisons of 360-degree novel views among calibration methods. Our results outperform in both rendering quality and camera accuracy. $\dag$ indicates training from scratch.
• Figure 5: Performance with different camera perturbations (PSNR$\uparrow$). Zoom in for details.