Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Gaussian Splatting

François Darmon, Lorenzo Porzi, Samuel Rota-Bulò, Peter Kontschieder

TL;DR

The paper identifies robustness gaps in 3D Gaussian Splatting for real-world handheld captures and proposes a unified framework to address blur, pose noise, and color drift. It models motion blur as a Gaussian distribution over poses, adds defocus blur covariance, and introduces a per-image affine RGB decoder, all integrated into the 3DGS representation while preserving training and rendering efficiency. Per-image parameters for motion, defocus, and color enable test-time pose/color adaptation, yielding improved results on challenging benchmarks. The approach achieves consistent gains over baselines on Scannet++ and Deblur-NeRF, enabling more reliable novel view synthesis from noisy real-world data.

Abstract

In this paper, we address common error sources for 3D Gaussian Splatting (3DGS) including blur, imperfect camera poses, and color inconsistencies, with the goal of improving its robustness for practical applications like reconstructions from handheld phone captures. Our main contribution involves modeling motion blur as a Gaussian distribution over camera poses, allowing us to address both camera pose refinement and motion blur correction in a unified way. Additionally, we propose mechanisms for defocus blur compensation and for addressing color in-consistencies caused by ambient light, shadows, or due to camera-related factors like varying white balancing settings. Our proposed solutions integrate in a seamless way with the 3DGS formulation while maintaining its benefits in terms of training efficiency and rendering speed. We experimentally validate our contributions on relevant benchmark datasets including Scannet++ and Deblur-NeRF, obtaining state-of-the-art results and thus consistent improvements over relevant baselines.

Robust Gaussian Splatting

TL;DR

The paper identifies robustness gaps in 3D Gaussian Splatting for real-world handheld captures and proposes a unified framework to address blur, pose noise, and color drift. It models motion blur as a Gaussian distribution over poses, adds defocus blur covariance, and introduces a per-image affine RGB decoder, all integrated into the 3DGS representation while preserving training and rendering efficiency. Per-image parameters for motion, defocus, and color enable test-time pose/color adaptation, yielding improved results on challenging benchmarks. The approach achieves consistent gains over baselines on Scannet++ and Deblur-NeRF, enabling more reliable novel view synthesis from noisy real-world data.

Abstract

In this paper, we address common error sources for 3D Gaussian Splatting (3DGS) including blur, imperfect camera poses, and color inconsistencies, with the goal of improving its robustness for practical applications like reconstructions from handheld phone captures. Our main contribution involves modeling motion blur as a Gaussian distribution over camera poses, allowing us to address both camera pose refinement and motion blur correction in a unified way. Additionally, we propose mechanisms for defocus blur compensation and for addressing color in-consistencies caused by ambient light, shadows, or due to camera-related factors like varying white balancing settings. Our proposed solutions integrate in a seamless way with the 3DGS formulation while maintaining its benefits in terms of training efficiency and rendering speed. We experimentally validate our contributions on relevant benchmark datasets including Scannet++ and Deblur-NeRF, obtaining state-of-the-art results and thus consistent improvements over relevant baselines.
Paper Structure (24 sections, 16 equations, 6 figures, 3 tables, 1 algorithm)

This paper contains 24 sections, 16 equations, 6 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Brief review of Gaussian splatting
  4. Gaussian primitive.
  5. Splatting.
  6. Rendering.
  7. Improving Robustness of Gaussian splatting
  8. Robustness against pose errors and motion blur
  9. Implementation details.
  10. Robustness against defocus blur
  11. Robustness against color inconsistencies
  12. Summary
  13. Experiments
  14. Evaluating NVS from noisy captures
  15. Comparison with NVS baselines on real-world data
  16. ...and 9 more sections

Figures (6)

  • Figure 1: Hand-held phone captures (top left, from ScanNet++) can be challenging to reconstruct with 3D Gaussian Splatting (top center), due to inter-frame color inconsistencies (bottom left), motion blur and defocus blur (bottom right). We show how these factors can explicitly and easily be modeled in the 3D GS framework, leading to notably improved reconstruction results (top right).
  • Figure 2: In 3DGS, defocus blur (left), color inconsistencies (center) and motion blur (right) can be modeled as simple transformations applied to the 3D Gaussian primitives (Sec. \ref{['sec:method']}). This allows us to estimate per-camera motion, appearance and focus parameters, which can be factored out to recover a sharp reconstruction.
  • Figure 3: We perform test-time optimization of the per-image parameters to ensure a fair comparison. Without it, the renderings are noticeably misaligned and they have a color shift compared to the ground truth.
  • Figure 4: Qualitative comparison of GS models with different ablations.
  • Figure 5: Comparison of our approach with state-of-the-art Mipnerf360 mipnerf360 and Nerfacto nerfstudio. All the methods are trained with per-image affine color transformation and pose optimization followed by test-time optimization of the per-image parameters on the test views.
  • ...and 1 more figures