VarSplat: Uncertainty-aware 3D Gaussian Splatting for Robust RGB-D SLAM

TL;DR

VarSplat is introduced, an uncertainty-aware 3DGS-SLAM system that explicitly learns per-splat appearance variance and renders differentiable per-pixel uncertainty map via efficient, single-pass rasterization.

Abstract

Simultaneous Localization and Mapping (SLAM) with 3D Gaussian Splatting (3DGS) enables fast, differentiable rendering and high-fidelity reconstruction across diverse real-world scenes. However, existing 3DGS-SLAM approaches handle measurement reliability implicitly, making pose estimation and global alignment susceptible to drift in low-texture regions, transparent surfaces, or areas with complex reflectance properties. To this end, we introduce VarSplat, an uncertainty-aware 3DGS-SLAM system that explicitly learns per-splat appearance variance. By using the law of total variance with alpha compositing, we then render differentiable per-pixel uncertainty map via efficient, single-pass rasterization. This map guides tracking, submap registration, and loop detection toward focusing on reliable regions and contributes to more stable optimization. Experimental results on Replica (synthetic) and TUM-RGBD, ScanNet, and ScanNet++ (real-world) show that VarSplat improves robustness and achieves competitive or superior tracking, mapping, and novel view synthesis rendering compared to existing studies for dense RGB-D SLAM.

Figures (5)

• Figure 1: VarSplat. Given RGB-D inputs, each 3D Gaussian jointly learns position, orientation, scale, color, opacity, and appearance variance $\sigma^2$. During mapping, $\sigma^2$ is optimized jointly with other Gaussian parameters. The rendered per-pixel uncertainty $V$ serves as a confidence weight during tracking and registration, while the per-Gaussian variance $\sigma^2$ further guides loop detection, enabling robust and uncertainty-aware SLAM.
• Figure 2: VarSplat architecture. During mapping, each 3D Gaussian jointly learns position, appearance, and variance $\sigma^2$. The per-splat variances are composited into per-pixel uncertainty $V$ through the differentiable rasterizer in mapping. This uncertainty map is then served as confidence in tracking and registration, while $\sigma^2$ is used for uncertainty-aware loop detection.
• Figure 3: Uncertainty ablation on ScanNet (scene0181). Without uncertainty, tracking jitters, loop detection has long-range drift, and registration ghosts submaps. With VarSplat enabled, the trajectory is smooth and overlaps align.
• Figure 4: Visualization of challenging conditions (scene0169 ScanNet). In the top example, uncertainty is high on the texture-poor wall region (white box), while the map continues to grow (blue box). In the bottom example, the uncertainty map can distinguish reliable regions on transparent surface.
• Figure 5: Per-pixel uncertainty with vs. without depth on TUM-RGBD (fr1/desk2). With depth, uncertainty focuses on textureless areas with depth holes and stays low on well constrained surfaces, avoiding overconfidence on glossy areas.