Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Temporal-Consistent Video Restoration with Pre-trained Diffusion Models

Hengkang Wang, Yang Liu, Huidong Liu, Chien-Chih Wang, Yanhui Guo, Hongdong Li, Bryan Wang, Ju Sun

TL;DR

This work tackles video restoration with pre-trained image diffusion models by reformulating the problem as a Maximum a Posteriori (MAP) optimization in the seed space of diffusion models. By reparameterizing frames as X = D ∘ R(Z) and optimizing seeds Z, the authors avoid accumulation of diffusion-approximation errors and leverage strong LDM priors for VR. They introduce a bilevel temporal-consistency strategy: semantic-level consistency via a noise-prior clustering of seeds (Z = z_shared 1^T + R) and pixel-level consistency via progressive warping with optical-flow refinements (RAFT) and EMA-smoothed flows, plus an efficient optimization framework using DDIM with only 4 steps, a mean-value theorem reformulation, and low-rank residuals to reduce memory. Experiments across five VR tasks on DAVIS, REDS, SPMS, and UDM10 demonstrate substantial improvements in both visual quality (PSNR/SSIM/LPIPS) and temporal consistency (WE), with notable robustness to diverse degradations and better scalability than CG-based approaches.

Abstract

Video restoration (VR) aims to recover high-quality videos from degraded ones. Although recent zero-shot VR methods using pre-trained diffusion models (DMs) show good promise, they suffer from approximation errors during reverse diffusion and insufficient temporal consistency. Moreover, dealing with 3D video data, VR is inherently computationally intensive. In this paper, we advocate viewing the reverse process in DMs as a function and present a novel Maximum a Posterior (MAP) framework that directly parameterizes video frames in the seed space of DMs, eliminating approximation errors. We also introduce strategies to promote bilevel temporal consistency: semantic consistency by leveraging clustering structures in the seed space, and pixel-level consistency by progressive warping with optical flow refinements. Extensive experiments on multiple virtual reality tasks demonstrate superior visual quality and temporal consistency achieved by our method compared to the state-of-the-art.

Temporal-Consistent Video Restoration with Pre-trained Diffusion Models

TL;DR

This work tackles video restoration with pre-trained image diffusion models by reformulating the problem as a Maximum a Posteriori (MAP) optimization in the seed space of diffusion models. By reparameterizing frames as X = D ∘ R(Z) and optimizing seeds Z, the authors avoid accumulation of diffusion-approximation errors and leverage strong LDM priors for VR. They introduce a bilevel temporal-consistency strategy: semantic-level consistency via a noise-prior clustering of seeds (Z = z_shared 1^T + R) and pixel-level consistency via progressive warping with optical-flow refinements (RAFT) and EMA-smoothed flows, plus an efficient optimization framework using DDIM with only 4 steps, a mean-value theorem reformulation, and low-rank residuals to reduce memory. Experiments across five VR tasks on DAVIS, REDS, SPMS, and UDM10 demonstrate substantial improvements in both visual quality (PSNR/SSIM/LPIPS) and temporal consistency (WE), with notable robustness to diverse degradations and better scalability than CG-based approaches.

Abstract

Video restoration (VR) aims to recover high-quality videos from degraded ones. Although recent zero-shot VR methods using pre-trained diffusion models (DMs) show good promise, they suffer from approximation errors during reverse diffusion and insufficient temporal consistency. Moreover, dealing with 3D video data, VR is inherently computationally intensive. In this paper, we advocate viewing the reverse process in DMs as a function and present a novel Maximum a Posterior (MAP) framework that directly parameterizes video frames in the seed space of DMs, eliminating approximation errors. We also introduce strategies to promote bilevel temporal consistency: semantic consistency by leveraging clustering structures in the seed space, and pixel-level consistency by progressive warping with optical flow refinements. Extensive experiments on multiple virtual reality tasks demonstrate superior visual quality and temporal consistency achieved by our method compared to the state-of-the-art.

Paper Structure

This paper contains 28 sections, 12 equations, 9 figures, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and related work
  3. Diffusion models (DMs)
  4. DMs for video restoration
  5. Our method
  6. Our basic formulation
  7. Promoting bilevel temporal consistency
  8. Noise prior for semantic-level consistency
  9. An intriguing clustering phenomenon
  10. Progressive warping for pixel-level consistency
  11. Boosting computational and memory efficiency
  12. Efficient diffusion sampling
  13. Efficient reformulation via mean value theorem (MVT)
  14. Lightweight low-rank residual parameterization
  15. Experiments
  16. ...and 13 more sections

Figures (9)

  • Figure 1: Visualization of sample VR results by our method: (a) Super-resolution $\times 4$; (b) Inpainting with $50\%$ random pixel masking; (c) Temporal deconvolution using uniform PSF with kernel width $k = 7$; and (d) Temporal deconvolution with motion deblurring.
  • Figure 2: T-SNE maaten_visualizing_2008 visualization of seeds extracted for video frames. Green dots are i.i.d. Gaussian noise. Seeds for the same video form clusters, while those for different videos are scattered.
  • Figure 3: Evolution of key metrics during the iterative VR process: PSNR, OF difference (measured against ground truth), and warping error (WE). All metrics improve in early iterations. After OF difference stabilizes (vertical dotted line), progressive warping is activated, further enhancing PSNR and reducing WE, demonstrating our framework's effectiveness for temporal consistency.
  • Figure 4: Qualitative comparisons on the DAVIS dataset: (a) Super-resolution $\times 4$, (b) Inpainting with $50\%$ random pixel masking, (c) Temporal deconvolution using uniform PSF with kernel width $k = 7$, and (d) Temporal deconvolution with spatial deblurring.
  • Figure 5: Qualitative comparisons for video super-resolution $\times 4$
  • ...and 4 more figures