DiET-GS: Diffusion Prior and Event Stream-Assisted Motion Deblurring 3D Gaussian Splatting

TL;DR

DiET-GS addresses the challenge of reconstructing sharp 3D representations from blurry multi-view imagery by fusing blur-free event streams with a pretrained diffusion prior. It introduces a two-stage process: Stage 1 (DiET-GS) leverages Event Double Integral (EDI) constraints in RGB and brightness domains, a learnable CRF, and Renoised Score Distillation to constrain 3D Gaussian Splatting, while Stage 2 (DiET-GS++) adds per-Gaussian latent features to maximize diffusion guidance and further sharpen edges. The combined approach achieves superior novel-view synthesis quality on synthetic and real data, outperforming baselines in both perceptual and NR-IQA metrics, with wavelet-based color correction helping reconcile diffusion-induced color shifts. This method offers practical impact for high-quality 3D reconstruction under motion blur, especially in low-light or fast-motion settings, by effectively integrating event-based sensing and diffusion priors without retraining large diffusion models.

Abstract

Reconstructing sharp 3D representations from blurry multi-view images are long-standing problem in computer vision. Recent works attempt to enhance high-quality novel view synthesis from the motion blur by leveraging event-based cameras, benefiting from high dynamic range and microsecond temporal resolution. However, they often reach sub-optimal visual quality in either restoring inaccurate color or losing fine-grained details. In this paper, we present DiET-GS, a diffusion prior and event stream-assisted motion deblurring 3DGS. Our framework effectively leverages both blur-free event streams and diffusion prior in a two-stage training strategy. Specifically, we introduce the novel framework to constraint 3DGS with event double integral, achieving both accurate color and well-defined details. Additionally, we propose a simple technique to leverage diffusion prior to further enhance the edge details. Qualitative and quantitative results on both synthetic and real-world data demonstrate that our DiET-GS is capable of producing significantly better quality of novel views compared to the existing baselines. Our project page is https://diet-gs.github.io

Figures (11)

• Figure 1: Overall framework of our DiET-GS.Stage 1 (DiET-GS) optimizes the deblurring 3DGS with the event streams and diffusion prior. To preserve accurate color and clean details, we exploit the EDI prior in multiple ways, including color supervision $C$, guidance for fine-grained details $I$, and additional regularization $\tilde{I}$ with EDI simulation. Stage 2 (DiET-GS++) is then employed to maximize the effect of diffusion prior by introducing extra learnable parameters $\mathbf{f}_{\mathbf{g}}$. DiET-GS++ further refines the rendered images from DiET-GS, effectively enhancing rich edge features. More details are explained in Sec. \ref{['sec:stage1']} and Sec. \ref{['sec:stage2']}.
• Figure 2: Cycle consistency among the objective terms.$\mathcal{L}_{\mathrm{edi\_simul}}$ follows the formulation of $\mathcal{L}_{\mathrm{edi\_gray}}$ except for substituting $\mathbf{C}^{\mathbf{B}}$ to simulated blurry image $\hat{\mathbf{C}}^{\mathbf{B}}$ derived from $\mathcal{L}_{\mathrm{blur}}$. It completes the cycle among the objective terms, further regularizing the fine-grained deblurring as shown in Fig. \ref{['fig:edi_simul_ablation']}.
• Figure 3: Qualitative comparisons on both synthetic (1st-2nd rows) and real-world (3rd-4th rows) datasets. DiET-GS shows cleaner texture with more accurate details compared to the event-based baselines while DiET-GS++ further enhances these features with sharper definition, achieving the best visual quality.
• Figure 4: Ablation study on $\mathcal{L}_{\mathrm{edi\_gray}}$ and $\mathcal{L}_{\mathrm{edi\_color}}$
• Figure 5: Ablation on $\mathcal{L}_{\mathrm{edi\_simul}}$ (1st row) and $\mathcal{L}_{\mathrm{rsd}}$ (S1) (2nd row).