Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zero-Shot Video Restoration and Enhancement Using Pre-Trained Image Diffusion Model

Cong Cao, Huanjing Yue, Xin Liu, Jingyu Yang

TL;DR

Problem: diffusion-based zero-shot video restoration suffers temporal flicker due to lack of temporal modeling. Approach: ZVRD leverages a pre-trained image diffusion prior augmented with Short-Long-Range temporal attention, temporal consistency guidance, spatial-temporal noise sharing, and an early stopping sampling strategy to enforce temporal coherence without training. Contributions: first training-free framework for zero-shot video restoration using an image diffusion prior; novel SLR temporal attention with cross-neighbor-frame and self-corrected trajectory components; temporal consistency mechanisms and noise-sharing strategy; extensive experiments across six tasks showing consistent improvements. Significance: enables robust, plug-and-play video restoration/enhancement by reusing existing image priors with minimal overhead, broadening practical deployment.

Abstract

Diffusion-based zero-shot image restoration and enhancement models have achieved great success in various tasks of image restoration and enhancement. However, directly applying them to video restoration and enhancement results in severe temporal flickering artifacts. In this paper, we propose the first framework for zero-shot video restoration and enhancement based on the pre-trained image diffusion model. By replacing the spatial self-attention layer with the proposed short-long-range (SLR) temporal attention layer, the pre-trained image diffusion model can take advantage of the temporal correlation between frames. We further propose temporal consistency guidance, spatial-temporal noise sharing, and an early stopping sampling strategy to improve temporally consistent sampling. Our method is a plug-and-play module that can be inserted into any diffusion-based image restoration or enhancement methods to further improve their performance. Experimental results demonstrate the superiority of our proposed method. Our code is available at https://github.com/cao-cong/ZVRD.

Zero-Shot Video Restoration and Enhancement Using Pre-Trained Image Diffusion Model

TL;DR

Problem: diffusion-based zero-shot video restoration suffers temporal flicker due to lack of temporal modeling. Approach: ZVRD leverages a pre-trained image diffusion prior augmented with Short-Long-Range temporal attention, temporal consistency guidance, spatial-temporal noise sharing, and an early stopping sampling strategy to enforce temporal coherence without training. Contributions: first training-free framework for zero-shot video restoration using an image diffusion prior; novel SLR temporal attention with cross-neighbor-frame and self-corrected trajectory components; temporal consistency mechanisms and noise-sharing strategy; extensive experiments across six tasks showing consistent improvements. Significance: enables robust, plug-and-play video restoration/enhancement by reusing existing image priors with minimal overhead, broadening practical deployment.

Abstract

Diffusion-based zero-shot image restoration and enhancement models have achieved great success in various tasks of image restoration and enhancement. However, directly applying them to video restoration and enhancement results in severe temporal flickering artifacts. In this paper, we propose the first framework for zero-shot video restoration and enhancement based on the pre-trained image diffusion model. By replacing the spatial self-attention layer with the proposed short-long-range (SLR) temporal attention layer, the pre-trained image diffusion model can take advantage of the temporal correlation between frames. We further propose temporal consistency guidance, spatial-temporal noise sharing, and an early stopping sampling strategy to improve temporally consistent sampling. Our method is a plug-and-play module that can be inserted into any diffusion-based image restoration or enhancement methods to further improve their performance. Experimental results demonstrate the superiority of our proposed method. Our code is available at https://github.com/cao-cong/ZVRD.
Paper Structure (21 sections, 11 equations, 4 figures, 5 tables)

This paper contains 21 sections, 11 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Diffusion-Based Zero-Shot Image Restoration and Enhancement
  4. Diffusion-Based Zero-Shot Video Editing
  5. Video Restoration and Enhancement
  6. Background
  7. The Reverse Diffusion Process
  8. Method
  9. Overall Framework
  10. SLR Temporal Attention
  11. Cross-Neighbor-Frame Attention
  12. Self-Corrected Trajectory Attention
  13. Temporal Consistency Guidance
  14. Spatial-Temporal Noise Sharing
  15. Early Stopping Sampling Strategy
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Framework of the proposed zero-shot video restoration and enhancement.
  • Figure 2: Architecture of the proposed SLR Temporal Attention. (a) The two modules in SLR temporal attention: cross-neighbor-frame attention and self-corrected trajectory attention, focus on short-range and long-range temporal correlation between frames, respectively. (b) The cross-neighbor-frame attention is applied first, and its output serves as the query for the self-corrected trajectory attention. (c) The procedure of self-corrected trajectory sampling. The red, yellow, and green trajectories denote the flow-based, similarity-based, and historically-best trajectories, respectively.
  • Figure 3: Visual quality comparison for video super-resolution. Zoom in for better observation.
  • Figure 4: Visual quality comparison for low-light video enhancement. Zoom in for better observation.