Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Modeling uncertainty for Gaussian Splatting

Luca Savant, Diego Valsesia, Enrico Magli

TL;DR

Stochastic Gaussian splatting is presented as the first framework for uncertainty estimation using Gaussian splatting, and a variational inference (VI)-based approach that seamlessly integrates uncertainty prediction into the common rendering pipeline of GS is introduced.

Abstract

We present Stochastic Gaussian Splatting (SGS): the first framework for uncertainty estimation using Gaussian Splatting (GS). GS recently advanced the novel-view synthesis field by achieving impressive reconstruction quality at a fraction of the computational cost of Neural Radiance Fields (NeRF). However, contrary to the latter, it still lacks the ability to provide information about the confidence associated with their outputs. To address this limitation, in this paper, we introduce a Variational Inference-based approach that seamlessly integrates uncertainty prediction into the common rendering pipeline of GS. Additionally, we introduce the Area Under Sparsification Error (AUSE) as a new term in the loss function, enabling optimization of uncertainty estimation alongside image reconstruction. Experimental results on the LLFF dataset demonstrate that our method outperforms existing approaches in terms of both image rendering quality and uncertainty estimation accuracy. Overall, our framework equips practitioners with valuable insights into the reliability of synthesized views, facilitating safer decision-making in real-world applications.

Modeling uncertainty for Gaussian Splatting

TL;DR

Stochastic Gaussian splatting is presented as the first framework for uncertainty estimation using Gaussian splatting, and a variational inference (VI)-based approach that seamlessly integrates uncertainty prediction into the common rendering pipeline of GS is introduced.

Abstract

We present Stochastic Gaussian Splatting (SGS): the first framework for uncertainty estimation using Gaussian Splatting (GS). GS recently advanced the novel-view synthesis field by achieving impressive reconstruction quality at a fraction of the computational cost of Neural Radiance Fields (NeRF). However, contrary to the latter, it still lacks the ability to provide information about the confidence associated with their outputs. To address this limitation, in this paper, we introduce a Variational Inference-based approach that seamlessly integrates uncertainty prediction into the common rendering pipeline of GS. Additionally, we introduce the Area Under Sparsification Error (AUSE) as a new term in the loss function, enabling optimization of uncertainty estimation alongside image reconstruction. Experimental results on the LLFF dataset demonstrate that our method outperforms existing approaches in terms of both image rendering quality and uncertainty estimation accuracy. Overall, our framework equips practitioners with valuable insights into the reliability of synthesized views, facilitating safer decision-making in real-world applications.
Paper Structure (15 sections, 18 equations, 2 figures, 2 tables)

This paper contains 15 sections, 18 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. NeRF and Gaussian Splatting
  4. Structure from Motion uncertainty estimation
  5. Radiance Field and Direct Volume Rendering
  6. Method
  7. Stochastic Gaussian Splatting
  8. Learning with Variational Inference
  9. Learning with AUSE
  10. End-to-end SGS Training
  11. Experimental results
  12. Experimental Setting
  13. Main Results
  14. AUSE Loss Ablation
  15. Conclusions

Figures (2)

  • Figure 1: Bayesian Network Graphical Model of SGS. Learnable variables are depicted in blue, while stochastic variables are circled. The "Camera" node represents both the spatial coordinate of the pixel $(\mathbf{o}, \mathbf{d})$ and the corresponding camera intrinsic and extrinsic parameters. Gray dashed rectangles are used for the plate notation, i.e. variables repetitions.
  • Figure 2: A qualitative example of our method SGS with CF-NeRF shen2022conditional. The last column is a visualization of the predicted uncertainty map.