DOVE: Efficient One-Step Diffusion Model for Real-World Video Super-Resolution

TL;DR

This work tackles real-world video super-resolution with diffusion models, where multi-step sampling limits practical deployment. It introduces DOVE, a one-step diffusion model fine-tuned from the pretrained CogVideoX, coupled with a two-stage latent-pixel training strategy and a curated HQ-VSR dataset crafted via a dedicated video-processing pipeline. Empirical results show DOVE achieves state-of-the-art performance across multiple benchmarks while delivering up to 28x faster inference than prior diffusion-based VSR methods. The combination of a strong pretrained video prior, an efficient training regimen, and high-quality data makes diffusion-based VSR viable for real-world applications with rigorous fidelity and temporal consistency demands.

Abstract

Diffusion models have demonstrated promising performance in real-world video super-resolution (VSR). However, the dozens of sampling steps they require, make inference extremely slow. Sampling acceleration techniques, particularly single-step, provide a potential solution. Nonetheless, achieving one step in VSR remains challenging, due to the high training overhead on video data and stringent fidelity demands. To tackle the above issues, we propose DOVE, an efficient one-step diffusion model for real-world VSR. DOVE is obtained by fine-tuning a pretrained video diffusion model (i.e., CogVideoX). To effectively train DOVE, we introduce the latent-pixel training strategy. The strategy employs a two-stage scheme to gradually adapt the model to the video super-resolution task. Meanwhile, we design a video processing pipeline to construct a high-quality dataset tailored for VSR, termed HQ-VSR. Fine-tuning on this dataset further enhances the restoration capability of DOVE. Extensive experiments show that DOVE exhibits comparable or superior performance to multi-step diffusion-based VSR methods. It also offers outstanding inference efficiency, achieving up to a 28$\times$ speed-up over existing methods such as MGLD-VSR. Code is available at: https://github.com/zhengchen1999/DOVE.

Figures (5)

• Figure 1: Efficiency and performance comparisons on the real-world benchmark (i.e., VideoLQ chan2022investigating). We provide qualitative (left) and quantitative (right) results. The running time (Time) is measured on one A100 GPU using a 33-frame 720$\times$1280 video. Our method achieves impressive performance and excellent efficiency. Compared with MGLD-VSR yang2024motion, DOVE is approximately 28$\times$ faster.
• Figure 2: Overview of the framework and training strategy of DOVE. Our method performs one-step sampling to reconstruct HR videos ($\mathbf{x}_{sr}$) from LR inputs ($\mathbf{x}_{lr}$). To enable effective training, we adopt the two-stage latent-pixel training strategy. Stage-1 (latent-space): Minimize the difference between the predicted and HR latents. Stage-2 (pixel-space): Improve detail generation using mixed image / video training, where the data branch at each iteration is controlled by image ratio ($\varphi$). To reduce memory cost, video is processed frame-by-frame in the encoder and decoder.
• Figure 3: The illustration of the video processing pipeline (four steps). Based on this pipeline, we construct HQ-VSR, a high-quality dataset tailored for the VSR task.
• Figure 4: Visual comparison on synthetic (YouHQ40 zhou2024upscale) and real-world (VideoLQ chan2022investigating) datasets. The videos in VideoLQ are sourced from the Internet without high-resolution (HQ) references.
• Figure 5: Comparison of temporal consistency (stacking the red line across frames).