Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IBGS: Image-Based Gaussian Splatting

Hoang Chuong Nguyen, Wei Mao, Jose M. Alvarez, Miaomiao Liu

TL;DR

This work proposes Image-Based Gaussian Splatting, an efficient alternative that leverages high-resolution source images for fine details and view-specific color modeling and significantly outperforms prior Gaussian Splatting approaches in rendering quality, without increasing the storage footprint.

Abstract

3D Gaussian Splatting (3DGS) has recently emerged as a fast, high-quality method for novel view synthesis (NVS). However, its use of low-degree spherical harmonics limits its ability to capture spatially varying color and view-dependent effects such as specular highlights. Existing works augment Gaussians with either a global texture map, which struggles with complex scenes, or per-Gaussian texture maps, which introduces high storage overhead. We propose Image-Based Gaussian Splatting, an efficient alternative that leverages high-resolution source images for fine details and view-specific color modeling. Specifically, we model each pixel color as a combination of a base color from standard 3DGS rendering and a learned residual inferred from neighboring training images. This promotes accurate surface alignment and enables rendering images of high-frequency details and accurate view-dependent effects. Experiments on standard NVS benchmarks show that our method significantly outperforms prior Gaussian Splatting approaches in rendering quality, without increasing the storage footprint.

IBGS: Image-Based Gaussian Splatting

TL;DR

This work proposes Image-Based Gaussian Splatting, an efficient alternative that leverages high-resolution source images for fine details and view-specific color modeling and significantly outperforms prior Gaussian Splatting approaches in rendering quality, without increasing the storage footprint.

Abstract

3D Gaussian Splatting (3DGS) has recently emerged as a fast, high-quality method for novel view synthesis (NVS). However, its use of low-degree spherical harmonics limits its ability to capture spatially varying color and view-dependent effects such as specular highlights. Existing works augment Gaussians with either a global texture map, which struggles with complex scenes, or per-Gaussian texture maps, which introduces high storage overhead. We propose Image-Based Gaussian Splatting, an efficient alternative that leverages high-resolution source images for fine details and view-specific color modeling. Specifically, we model each pixel color as a combination of a base color from standard 3DGS rendering and a learned residual inferred from neighboring training images. This promotes accurate surface alignment and enables rendering images of high-frequency details and accurate view-dependent effects. Experiments on standard NVS benchmarks show that our method significantly outperforms prior Gaussian Splatting approaches in rendering quality, without increasing the storage footprint.

Paper Structure

This paper contains 14 sections, 16 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Preliminary: 3D Gaussian Splatting
  5. Modeling Spatially Varying and View-Dependent Color
  6. Feature Extraction from Source Views
  7. Color Residual Prediction
  8. Exposure Correction
  9. Optimization
  10. Experiments
  11. Experimental Setup
  12. Results
  13. Ablation Studies
  14. Conclusion

Figures (3)

  • Figure 1: Our pipeline. The color of each pixel ($\mathbf{c}^{\text{final}}$) consists of two components: a base color $\mathbf{c}$ (in pink boxes) which follows the standard 3DGS rendering process and a color residual $\Delta\mathbf{c}$ predicted from the warped color of different source views $\mathbf{c}_{m}^{\text{warp}}$. While the figure shows an example of using only two source views, in practice, our method can process an arbitrary number of source views.
  • Figure 2: Qualitative results. Our method can render images with both high-frequency details (first two scenes) and view-dependent effect (last two scenes). However this cannot be achieved by 3DGS kerbl20233d and SuperGaussian xu2024supergaussians
  • Figure 3: Comparison of rendered images with and without exposure correction. Our method can correct the exposure in both underexposure (top) and overexposure (bottom) cases.