Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Image Motion Blur Removal in the Temporal Dimension with Video Diffusion Models

Wang Pang, Zhihao Zhan, Xiang Zhu, Yechao Bai

TL;DR

This work reframes motion blur as a temporal averaging process and uses a pre-trained video diffusion transformer within a Diffusion Posterior Sampling framework to recover latent sharp frames from a single blurred image. By compressing video content into a latent space with a VQ-GAN and operating the diffusion process on this latent representation, the method avoids explicit blur-kernel estimation and handles complex, non-linear motion. The approach, evaluated on synthetic CLEVRER data and real BAIR footage, demonstrates superior deblurring performance over kernel-based and single-image baselines and shows robustness to mismatches in the blur formation model. The results highlight the potential of video diffusion priors as powerful world models for single-image motion deblurring and point toward scalable deployment on large diffusion platforms for broader applicability.

Abstract

Most motion deblurring algorithms rely on spatial-domain convolution models, which struggle with the complex, non-linear blur arising from camera shake and object motion. In contrast, we propose a novel single-image deblurring approach that treats motion blur as a temporal averaging phenomenon. Our core innovation lies in leveraging a pre-trained video diffusion transformer model to capture diverse motion dynamics within a latent space. It sidesteps explicit kernel estimation and effectively accommodates diverse motion patterns. We implement the algorithm within a diffusion-based inverse problem framework. Empirical results on synthetic and real-world datasets demonstrate that our method outperforms existing techniques in deblurring complex motion blur scenarios. This work paves the way for utilizing powerful video diffusion models to address single-image deblurring challenges.

Image Motion Blur Removal in the Temporal Dimension with Video Diffusion Models

TL;DR

This work reframes motion blur as a temporal averaging process and uses a pre-trained video diffusion transformer within a Diffusion Posterior Sampling framework to recover latent sharp frames from a single blurred image. By compressing video content into a latent space with a VQ-GAN and operating the diffusion process on this latent representation, the method avoids explicit blur-kernel estimation and handles complex, non-linear motion. The approach, evaluated on synthetic CLEVRER data and real BAIR footage, demonstrates superior deblurring performance over kernel-based and single-image baselines and shows robustness to mismatches in the blur formation model. The results highlight the potential of video diffusion priors as powerful world models for single-image motion deblurring and point toward scalable deployment on large diffusion platforms for broader applicability.

Abstract

Most motion deblurring algorithms rely on spatial-domain convolution models, which struggle with the complex, non-linear blur arising from camera shake and object motion. In contrast, we propose a novel single-image deblurring approach that treats motion blur as a temporal averaging phenomenon. Our core innovation lies in leveraging a pre-trained video diffusion transformer model to capture diverse motion dynamics within a latent space. It sidesteps explicit kernel estimation and effectively accommodates diverse motion patterns. We implement the algorithm within a diffusion-based inverse problem framework. Empirical results on synthetic and real-world datasets demonstrate that our method outperforms existing techniques in deblurring complex motion blur scenarios. This work paves the way for utilizing powerful video diffusion models to address single-image deblurring challenges.
Paper Structure (10 sections, 15 equations, 3 figures, 2 tables, 1 algorithm)

This paper contains 10 sections, 15 equations, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Diffusion Models
  4. Diffusion Posterior Sampling (DPS)
  5. Video Diffusion Models (VDMs)
  6. Proposed Approach
  7. Experiments
  8. Synthetic Dataset
  9. BAIR Dataset
  10. Conclusions

Figures (3)

  • Figure 1: Overview of the VDM-MD method: In the core iteration, the estimated 3D sharp video resides in the latent space, represented by green boxes. It is generated and refined by the pre-trained VDM, which includes several STDiT blocks. The latent video is then decoded and compared with the blurry image through the degradation model, indicated by red boxes. Their discrepancies are used to correct and enhance the video. Upon completion the latent video is decoded back to the visual space.
  • Figure 2: Motion deblurring examples with CLEVRER dataset. Each blurry inputs are generated by averaging 10 frames. Only the 0th, 3rd, 6th, and 9th frame of the GT and output videos are illustrated.
  • Figure 3: Comparison on BAIR dataset. For the GT and reconstructed videos only the 5th (middle) frame is shown.