PRIMU: Uncertainty Estimation for Novel Views in Gaussian Splatting from Primitive-Based Representations of Error and Coverage

TL;DR

<3-5 sentence high-level summary> PRIMU tackles unreliable uncertainty estimation in Gaussian Splatting by introducing primitive-based representations of training-view coverage and rendering error. It renders these representations as uncertainty feature maps and learns per-pixel uncertainty through lightweight gradient-boosting regression using a single hold-out view, enabling accurate depth and RGB UE without retraining GS. The framework supports direction-dependent, view-aware uncertainty and demonstrates strong correlations to true errors, especially for foreground objects, while also guiding active view selection. Across diverse datasets, PRIMU achieves state-of-the-art UE and shows robust generalization to unseen scenes, with clear benefits from foreground-focused analysis and a full feature-map set.

Abstract

We introduce Primitive-based Representations of Uncertainty (PRIMU), a post-hoc uncertainty estimation (UE) framework for Gaussian Splatting (GS). Reliable UE is essential for deploying GS in safety-critical domains such as robotics and medicine. Existing approaches typically estimate Gaussian-primitive variances and rely on the rendering process to obtain pixel-wise uncertainties. In contrast, we construct primitive-level representations of error and visibility/coverage from training views, capturing interpretable uncertainty information. These representations are obtained by projecting view-dependent training errors and coverage statistics onto the primitives. Uncertainties for novel views are inferred by rendering these primitive-level representations, producing uncertainty feature maps, which are aggregate through pixel-wise regression on holdout data. We analyze combinations of uncertainty feature maps and regression models to understand how their interactions affect prediction accuracy and generalization. PRIMU also enables an effective active view selection strategy by directly leveraging these uncertainty feature maps. Additionally, we study the effect of separating splatting into foreground and background regions. Our estimates show strong correlations with true errors, outperforming state-of-the-art methods, especially for depth UE and foreground objects. Finally, our regression models show generalization capabilities to unseen scenes, enabling UE without additional holdout data.

Figures (9)

• Figure 1: Comparison of PRIMU uncertainty maps to error for color and depth renderings. Undefined depth error is represented as light gray. Additional examples in Appendix \ref{['appen:VisualExamples']}.
• Figure 2: PRIMU projects rendering errors, observation coverage, and field-of-view statistics from training views onto Gaussian primitives. From these, it creates Gaussian primitive representations that can be rendered from novel viewpoints to produce uncertainty feature maps. Pixel-wise regression then predicts per-pixel errors as uncertainty estimates. Visual examples are shown in \ref{['fig:page1fig']} and Appendix \ref{['appen:VisualExamples']}.
• Figure 3: Qualitative comparison of uncertainty feature maps of FisherRF Jiang2023FisherRF, manifold lyu2024manifold, var3DGS li2024variational and direction-dependent PRIMU* (ours), to the $\ell_1$ rendering error. Scenes from top to bottom: MipNeRF360 garden and kitchen, LLFF fern and LF basket.
• Figure 4: Step-wise backward regression results identifying the most informative uncertainty feature maps. Left: Pearson correlation between predicted and true depth error over the number of feature maps used. Right: average feature map retention times.
• Figure 5: Average in-scene Pearson correlations of predicted and true rendering (left) and depth errors (right) for CNN models with varying receptive field sizes and numbers of uncertainty features.