DirectFisheye-GS: Enabling Native Fisheye Input in Gaussian Splatting with Cross-View Joint Optimization

Abstract

3D Gaussian Splatting (3DGS) has enabled efficient 3D scene reconstruction from everyday images with real-time, high-fidelity rendering, greatly advancing VR/AR applications. Fisheye cameras, with their wider field of view (FOV), promise high-quality reconstructions from fewer inputs and have recently attracted much attention. However, since 3DGS relies on rasterization, most subsequent works involving fisheye camera inputs first undistort images before training, which introduces two problems: 1) Black borders at image edges cause information loss and negate the fisheye's large FOV advantage; 2) Undistortion's stretch-and-interpolate resampling spreads each pixel's value over a larger area, diluting detail density -- causes 3DGS overfitting these low-frequency zones, producing blur and floating artifacts. In this work, we integrate fisheye camera model into the original 3DGS framework, enabling native fisheye image input for training without preprocessing. Despite correct modeling, we observed that the reconstructed scenes still exhibit floaters at image edges: Distortion increases toward the periphery, and 3DGS's original per-iteration random-selecting-view optimization ignores the cross-view correlations of a Gaussian, leading to extreme shapes (e.g., oversized or elongated) that degrade reconstruction quality. To address this, we introduce a feature-overlap-driven cross-view joint optimization strategy that establishes consistent geometric and photometric constraints across views-a technique equally applicable to existing pinhole-camera-based pipelines. Our DirectFisheye-GS matches or surpasses state-of-the-art performance on public datasets.

Figures (10)

• Figure 1: DirectFisheye-GS enables native fisheye image input for 3DGS training, avoiding information loss caused by undistortion. It achieves, and in many cases surpasses, state-of-the-art performance on various public datasets, demonstrating superior detail preservation and cross-view geometric & global illumination consistency.
• Figure 2: Common fisheye camera projection models. It is typically approximated as a unit sphere projection model. $P$ represents a Gaussian point in space, $C$ is the camera center, $\theta$ is the angle between the ray $PC$ and the camera optical axis, $f$ is the camera focal length, and $\mathbf r_d$ is the distance from the planar projection point $p$ to the image center.
• Figure 3: Illustration of the 3DGS single-view training paradigm and our proposed cross-view joint optimization strategy.
• Figure 4: Schematic diagram of our camera association method.
• Figure 5: Qualitative comparison of our results against the baselines on the FisheyeNeRF jeong2021self dataset. Ours has more details, better texture, and fewer floaters.