DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models

TL;DR

DiffIR2VR-Zero tackles zero-shot video restoration by repurposing pre-trained image restoration diffusion models without any training. It introduces hierarchical latent warping and hybrid flow-guided token merging to maintain temporal coherence while preserving detail. Extensive experiments demonstrate state-of-the-art results across denoising and super-resolution tasks, including 8× SR and high-noise scenarios, with near-training-level quality and strong robustness. The approach is model-agnostic and generalizes to other video tasks like depth estimation, offering a practical, training-free route to high-quality, temporally-consistent video enhancement.

Abstract

We present DiffIR2VR-Zero, a zero-shot framework that enables any pre-trained image restoration diffusion model to perform high-quality video restoration without additional training. While image diffusion models have shown remarkable restoration capabilities, their direct application to video leads to temporal inconsistencies, and existing video restoration methods require extensive retraining for different degradation types. Our approach addresses these challenges through two key innovations: a hierarchical latent warping strategy that maintains consistency across both keyframes and local frames, and a hybrid token merging mechanism that adaptively combines optical flow and feature matching. Through extensive experiments, we demonstrate that our method not only maintains the high-quality restoration of base diffusion models but also achieves superior temporal consistency across diverse datasets and degradation conditions, including challenging scenarios like 8$\times$ super-resolution and severe noise. Importantly, our framework works with any image restoration diffusion model, providing a versatile solution for video enhancement without task-specific training or modifications.

Figures (16)

• Figure 1: 4$\times$ video super-resolution results. (a) Traditional regression-based methods such as FMA-Net youk2024fmanet are limited to the training data domain and tend to produce blurry results when encountering out-of-domain inputs. (b) Although applying image-based diffusion models such as DiffBIR lin2024diffbir to individual frames can generate realistic details, these details often lack consistency across frames. (c) Our method leverages an image diffusion model to restore videos, achieving both realistic and consistent results without any additional training.
• Figure 2: Pipeline of our proposed zero-shot video restoration method. We process low-quality (LQ) videos in batches using a diffusion model, with a keyframe randomly sampled within each batch. (a) At the beginning of the diffusion denoising process, hierarchical latent warping provides rough shape guidance both globally, through latent warping between keyframes, and locally, by propagating these latents within the batch. (b) Throughout most of the denoising process, tokens are merged before the self-attention layer. For the downsample blocks, optical flow is used to find the correspondence between tokens, and for the upsample blocks, cosine similarity is utilized. This hybrid flow-guided, spatial-aware token merging accurately identifies correspondences between tokens by leveraging both flow and spatial information, thereby enhancing overall consistency at the token level.
• Figure 3: An illustration of our key modules. Without requiring any training, these modules can achieve coherence across frames by enforcing temporal stability in both latent and token space. Hierarchical latent warping provides global and local shape guidance; Hybrid spatial-aware token merging before the self-attention layer improves temporal consistency by matching similar tokens using optical flow in the down blocks and cosine similarity in the up blocks of the UNet.
• Figure 4: Token correspondences (cosine similarity and optical flow) across denoising steps. Early on (e.g., step 10), optical flow guides better due to noisy latents. Later (e.g., step 40), similarity and flow focus on different regions, showcasing the benefit of our hybrid approach for effective token merging throughout denoising.
• Figure 5: Qualitative comparisons on 4$\times$ video super-resolution. As shown in the first row, the low-quality input lacks almost all details. In the zoomed-in patches, our method produces clearer and more consistent results.