Self-Calibrating Gaussian Splatting for Large Field of View Reconstruction

TL;DR

This paper tackles the challenge of reconstructing wide-field scenes from uncalibrated, highly distorted fisheye imagery. It introduces Self-Calibrating Gaussian Splatting, a differentiable pipeline that jointly optimizes camera intrinsics, extrinsics, lens distortion, and 3D Gaussian scene representations, all within an end-to-end framework. A hybrid distortion field (combining invertible residual networks with an explicit control grid) together with cubemap-based resampling enables accurate, artifact-free reconstruction across the full field of view, without pre-calibration or restrictive projection models. The approach achieves state-of-the-art performance on synthetic and real-world data, demonstrates robust reconstruction with few input views, and provides broad applicability across diverse wide-angle lenses, making wide-FOV NVS more practical for applications in robotics, virtual reality, and autonomous systems.

Abstract

In this paper, we present a self-calibrating framework that jointly optimizes camera parameters, lens distortion and 3D Gaussian representations, enabling accurate and efficient scene reconstruction. In particular, our technique enables high-quality scene reconstruction from Large field-of-view (FOV) imagery taken with wide-angle lenses, allowing the scene to be modeled from a smaller number of images. Our approach introduces a novel method for modeling complex lens distortions using a hybrid network that combines invertible residual networks with explicit grids. This design effectively regularizes the optimization process, achieving greater accuracy than conventional camera models. Additionally, we propose a cubemap-based resampling strategy to support large FOV images without sacrificing resolution or introducing distortion artifacts. Our method is compatible with the fast rasterization of Gaussian Splatting, adaptable to a wide variety of camera lens distortion, and demonstrates state-of-the-art performance on both synthetic and real-world datasets.

Figures (21)

• Figure 1: We introduce Self-Calibrating Gaussian Splatting, a differentiable rasterization pipeline with a hybrid lens distortion field that can produce high-quality novel view synthesis results from uncalibrated wide-angle photographs. (a) Existing methods such as Fisheye-GS liao2024fisheye fail to accurately handle complex lens distortions due to the traditional parametric distortion model. (b) Our method accurately models large distortions, especially in the peripheral regions, utilizing the entire highly distorted raw images for reconstruction. (d) Our method (bottom) provides extensive coverage, whereas conventional pipelines (top) can only recover the center.
• Figure 2: Conventional Paradigm vs. Our Method. (a) Conventional approaches require reprojecting the image into perspective views compatible with 3DGS rasterization. As the field of view increases, pixel stretching becomes progressively severe, significantly compromising the quality of the reconstruction. (b) In contrast, our cubemap resampling strategy maintains a consistent pixel density across the entire field of view. This approach, combined with our hybrid distortion field, utilizes the peripheral regions (the annular area outside the blue box) without severe distortion or pixel stretching. Moreover, our method can handle fields of view up to 180°, as demonstrated by the green box, allowing for comprehensive and accurate reconstructions.
• Figure 3: Overview of Our Method. In contrast to the explicit distortion vector field illustrated in the upper row, our hybrid approach maintains computational efficiency by leveraging explicit control points. Additionally, the regularization provided by the invertible neural field effectively balances the trade-off between the expressiveness and smoothness of the distortion field.
• Figure 4: Qualitative Comparisons with Baselines on the FisheyeNeRF Datasetjeong2021self. The images show comparisons across different scenes using two baselines (e.g., ADOP-GS ruckert2022adop and Fisheye-GS liao2024fisheye) and our method. PSNRs are computed for each patch.
• Figure 5: Qualitative Comparisons with Fisheye-GS liao2024fisheye. To validate our hybrid distortion modeling, we further compare our method with Fisheye-GS liao2024fisheye on larger FOV scenes, including real-world captures using 150° cameras (a–c) and simulations using a 180° camera (d–f) in Mitsuba jakob2010mitsuba. Our method successfully recovers details in peripheral regions, whereas Fisheye-GS liao2024fisheye struggles.