BAD-Gaussians: Bundle Adjusted Deblur Gaussian Splatting

TL;DR

A novel approach, named BAD-Gaussians (Bundle Adjusted Deblur Gaussian Splatting), which leverages explicit Gaussian representation and handles severe motion-blurred images with inaccurate camera poses to achieve high-quality scene reconstruction and enables real-time rendering capabilities.

Abstract

While neural rendering has demonstrated impressive capabilities in 3D scene reconstruction and novel view synthesis, it heavily relies on high-quality sharp images and accurate camera poses. Numerous approaches have been proposed to train Neural Radiance Fields (NeRF) with motion-blurred images, commonly encountered in real-world scenarios such as low-light or long-exposure conditions. However, the implicit representation of NeRF struggles to accurately recover intricate details from severely motion-blurred images and cannot achieve real-time rendering. In contrast, recent advancements in 3D Gaussian Splatting achieve high-quality 3D scene reconstruction and real-time rendering by explicitly optimizing point clouds as Gaussian spheres. In this paper, we introduce a novel approach, named BAD-Gaussians (Bundle Adjusted Deblur Gaussian Splatting), which leverages explicit Gaussian representation and handles severe motion-blurred images with inaccurate camera poses to achieve high-quality scene reconstruction. Our method models the physical image formation process of motion-blurred images and jointly learns the parameters of Gaussians while recovering camera motion trajectories during exposure time. In our experiments, we demonstrate that BAD-Gaussians not only achieves superior rendering quality compared to previous state-of-the-art deblur neural rendering methods on both synthetic and real datasets but also enables real-time rendering capabilities. Our project page and source code is available at https://lingzhezhao.github.io/BAD-Gaussians/

Figures (6)

• Figure 1: The pipeline of BAD-Gaussians. Our approach utilizes Gaussian representations to depict sharp 3D scenes derived from a series of motion-blurred images, along with their inaccurate poses and sparse point clouds from COLMAP, serving as the initialization for the Gaussians. Employing forward projection and differentiable Gaussian rasterization, we jointly optimize the Gaussians in the scene and the camera trajectory within exposure time, by backpropagating gradients from Gaussians to camera poses. Following the physical process of motion blur, we model motion-blurred images by averaging the virtual sharp images captured during the exposure time. These virtual camera poses are represented and interpolated using a continuous spline within the $\mathbf{SE}(3)$ space. The joint optimization of Gaussians and camera trajectories is achieved by minimizing the photometric loss between synthesized and actual blurry images.
• Figure 2: Qualitative novel view synthesis results of different methods with synthetic datasets. Despite being trained with ground truth poses (*), BAD-Gaussians outperforms Deblur-NeRF* and DP-NeRF* in recovering high-quality scenes from motion-blurred images with inaccurate camera poses, showcasing its superior performance.
• Figure 3: Qualitative novel view synthesis results of different methods with the real datasets. The experimental results demonstrate that our method achieves superior performance over prior methods on the real dataset as well. In contrast, BAD-NeRF yields poorer results when applied to real data and exhibits satisfactory performance only within synthetic datasets.
• Figure A: Qualitative deblurring results of different methods with synthetic datasets from MBA-VO mba-vo and Deblur-NeRF deblur-nerf. The scenes, from left to right, encompass ArchViz-high, Cozy2room, Factory, and Trolley. Despite being trained with ground truth poses (*), BAD-Gaussians outperforms Deblur-NeRF* and DP-NeRF* in recovering high-quality scenes from motion-blurred images with inaccurate camera poses, showcasing its superior performance.
• Figure B: Qualitative novel view synthesis results of different methods with the real datasets from Deblur-NeRF deblur-nerf. The scenes, from left to right, encompass Basket, Coffee, Girl, and Stair. The experimental results demonstrate that our method achieves superior performance over prior methods on the real dataset as well.