Deblurring 3D Gaussian Splatting

TL;DR

<3-5 sentence high-level summary> Deblurring 3D Gaussian Splatting introduces a real-time deblurring framework for 3D-Gaussian Splatting by using a compact MLP to modulate per-Gaussian covariances and means, modeling defocus and camera motion blur during training. The method preserves real-time inference by keeping the rasterization-based render pipeline unchanged at test time, while enabling sharp reconstructions from blurry inputs. To handle sparse training data, it densifies the point cloud with extra points and applies depth-aware pruning to preserve far-plane details. Experimental results on blur benchmarks show competitive image quality and superior SSIM, along with FPS exceeding 800, demonstrating practical viability for real-time 3D scene deblurring.

Abstract

Recent studies in Radiance Fields have paved the robust way for novel view synthesis with their photorealistic rendering quality. Nevertheless, they usually employ neural networks and volumetric rendering, which are costly to train and impede their broad use in various real-time applications due to the lengthy rendering time. Lately 3D Gaussians splatting-based approach has been proposed to model the 3D scene, and it achieves remarkable visual quality while rendering the images in real-time. However, it suffers from severe degradation in the rendering quality if the training images are blurry. Blurriness commonly occurs due to the lens defocusing, object motion, and camera shake, and it inevitably intervenes in clean image acquisition. Several previous studies have attempted to render clean and sharp images from blurry input images using neural fields. The majority of those works, however, are designed only for volumetric rendering-based neural radiance fields and are not straightforwardly applicable to rasterization-based 3D Gaussian splatting methods. Thus, we propose a novel real-time deblurring framework, Deblurring 3D Gaussian Splatting, using a small Multi-Layer Perceptron (MLP) that manipulates the covariance of each 3D Gaussian to model the scene blurriness. While Deblurring 3D Gaussian Splatting can still enjoy real-time rendering, it can reconstruct fine and sharp details from blurry images. A variety of experiments have been conducted on the benchmark, and the results have revealed the effectiveness of our approach for deblurring. Qualitative results are available at https://benhenryl.github.io/Deblurring-3D-Gaussian-Splatting/

Figures (16)

• Figure 1: Performance comparison to state-of-the-art deblurring NeRFs. Ours achieved a fast rendering speed ($>$ 800 FPS vs. 1 FPS) while maintaining competitive rendered image quality (the x-axis is represented in log scale).
• Figure 2: Our method's overall workflow. $\gamma(\cdot)$ denotes positional encoding, $\odot$ and $\oplus$ denotes hadamard product and averaging operation for each, and $x$, $r$, $s$ stand for position, quaternion, and scaling of 3D Gaussian respectively. $\otimes$ is an operator that implements $\delta r \odot r$, $\delta s \odot s$, and $\delta x + x$. Dotted arrows and dashed arrows describe the pipeline for modeling camera motion blur and modeling defocus blur, respectively at training time. Solid arrows show the process of rendering sharp images at the inference time. More details are explained at \ref{['sec:deblurring_3dgs']}.
• Figure 3: Comparison on densifying point clouds during training. Left: Example training view. Middle: Point cloud at 5,000 training iterations without adding points. Right: Point cloud at 5,000 training iterations with adding extra points at 2,500 iterations.
• Figure 4: Comparison to pruning 3D Gaussians. Left: Given 3D Gaussians. Middle: Applying the pruning method proposed by 3D-GS which removes 3D Gaussians with the single threshold ($t_p$). Right: Our pruning method that discards unnecessary 3D Gaussians with different thresholds based on their depth.
• Figure 5: Qualitative results on real camera motion blur dataset.