Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fisheye-GS: Lightweight and Extensible Gaussian Splatting Module for Fisheye Cameras

Zimu Liao, Siyan Chen, Rong Fu, Yi Wang, Zhongling Su, Hao Luo, Li Ma, Linning Xu, Bo Dai, Hengjie Li, Zhilin Pei, Xingcheng Zhang

TL;DR

This innovative method recalculates the projection transformation and its gradients for fisheye cameras and can be seamlessly integrated as a module into other efficient 3D rendering methods, emphasizing its extensibility, lightweight nature, and modular design.

Abstract

Recently, 3D Gaussian Splatting (3DGS) has garnered attention for its high fidelity and real-time rendering. However, adapting 3DGS to different camera models, particularly fisheye lenses, poses challenges due to the unique 3D to 2D projection calculation. Additionally, there are inefficiencies in the tile-based splatting, especially for the extreme curvature and wide field of view of fisheye lenses, which are crucial for its broader real-life applications. To tackle these challenges, we introduce Fisheye-GS.This innovative method recalculates the projection transformation and its gradients for fisheye cameras. Our approach can be seamlessly integrated as a module into other efficient 3D rendering methods, emphasizing its extensibility, lightweight nature, and modular design. Since we only modified the projection component, it can also be easily adapted for use with different camera models. Compared to methods that train after undistortion, our approach demonstrates a clear improvement in visual quality.

Fisheye-GS: Lightweight and Extensible Gaussian Splatting Module for Fisheye Cameras

TL;DR

This innovative method recalculates the projection transformation and its gradients for fisheye cameras and can be seamlessly integrated as a module into other efficient 3D rendering methods, emphasizing its extensibility, lightweight nature, and modular design.

Abstract

Recently, 3D Gaussian Splatting (3DGS) has garnered attention for its high fidelity and real-time rendering. However, adapting 3DGS to different camera models, particularly fisheye lenses, poses challenges due to the unique 3D to 2D projection calculation. Additionally, there are inefficiencies in the tile-based splatting, especially for the extreme curvature and wide field of view of fisheye lenses, which are crucial for its broader real-life applications. To tackle these challenges, we introduce Fisheye-GS.This innovative method recalculates the projection transformation and its gradients for fisheye cameras. Our approach can be seamlessly integrated as a module into other efficient 3D rendering methods, emphasizing its extensibility, lightweight nature, and modular design. Since we only modified the projection component, it can also be easily adapted for use with different camera models. Compared to methods that train after undistortion, our approach demonstrates a clear improvement in visual quality.
Paper Structure (33 sections, 17 equations, 5 figures, 3 tables)

This paper contains 33 sections, 17 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Novel View Synthesis
  4. Neural Radiance Fields (NeRF).
  5. 3D Gaussian Splatting
  6. Various Camera Models for Radiance Fields
  7. Preliminaries
  8. 3D Gaussian Splatting
  9. Definition.
  10. Transformation.
  11. Rendering.
  12. Equidistant Projection Model for Fisheye Cameras
  13. Methods
  14. Analysis of the Rendering Process of 3DGS
  15. The Modified Projection and Gradients of the Means
  16. ...and 18 more sections

Figures (5)

  • Figure 1: Our Fisheye-GS. We have directly trained our 3DGS3DGS model from images captured from fisheye cameras without undistortion to pinhole cameras. We then integrate our Fisheye-GS as a lightweight module within FlashGS feng2024flashgsefficient3dgaussian, an efficient rendering technique for 3DGS, to evaluate its visual quality and performance.
  • Figure 2: Analysis and Modification of 3DGS pipeline. To apply a fisheye camera for 3DGS, we derive the equidistant projection and its Jacobian matrix. Then we implement the projection as the red arrows. We have also adjusted the back-propagation illustrated in the purple arrows to align the modified projection of the fisheye cameras. Our module enables both training and rendering of 3DGS for fisheye cameras.
  • Figure 3: Qualitative comparison between Fisheye-GS and the baseline. The baseline struggles to render on edges and corners due to the clipping and interpolation from undistortion.
  • Figure 4: Result on the synthetic dataset, varying from scenes and FOVs.
  • Figure 5: Comparison between rendered images in fisheye cameras and panorama cameras from the scene "Office Night" trained from Scannet++ dataset.