Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion

TL;DR

This work addresses the latency bottleneck of diffusion-based video super-resolution by introducing Stream-DiffVSR, a strictly online, past-frame conditioning framework. It realizes a practical online VSR pipeline by distilling a 50-step denoiser to 4 steps, and by coupling this with Auto-regressive Temporal Guidance and a Temporal-aware Decoder that leverage motion-aligned information from previous frames. The approach delivers state-of-the-art perceptual and temporal quality while achieving unprecedented low latency and memory efficiency for diffusion-based VSR, enabling real-time applications such as video conferencing and AR/VR. The results demonstrate that diffusion priors can be harnessed for online VSR without prohibitive latency, marking a significant step toward deployable diffusion-based video restoration.

Abstract

Diffusion-based video super-resolution (VSR) methods achieve strong perceptual quality but remain impractical for latency-sensitive settings due to reliance on future frames and expensive multi-step denoising. We propose Stream-DiffVSR, a causally conditioned diffusion framework for efficient online VSR. Operating strictly on past frames, it combines a four-step distilled denoiser for fast inference, an Auto-regressive Temporal Guidance (ARTG) module that injects motion-aligned cues during latent denoising, and a lightweight temporal-aware decoder with a Temporal Processor Module (TPM) that enhances detail and temporal coherence. Stream-DiffVSR processes 720p frames in 0.328 seconds on an RTX4090 GPU and significantly outperforms prior diffusion-based methods. Compared with the online SOTA TMP, it boosts perceptual quality (LPIPS +0.095) while reducing latency by over 130x. Stream-DiffVSR achieves the lowest latency reported for diffusion-based VSR, reducing initial delay from over 4600 seconds to 0.328 seconds, thereby making it the first diffusion VSR method suitable for low-latency online deployment. Project page: https://jamichss.github.io/stream-diffvsr-project-page/

Figures (14)

• Figure 1: Overview of Auto-regressive Temporal-aware Decoder. Given the denoised latent and warped previous frame, our decoder enhances temporal consistency using temporal processor modules. This module aligns and fuses these features via interpolation, convolution, and weighted fusion, effectively stabilizing detail reconstruction when decoding into the final RGB frame.
• Figure 2: Training pipeline of Stream-DiffVSR. The training process consists of three sequential stages: (1) Distilling the denoising U-Net to reduce diffusion steps while maintaining perceptual quality with training objective (\ref{['unet_distill_math']}); (2) Training the Temporal Processor Module (TPM) within the decoder to enhance temporal consistency at the RGB level with training objective (\ref{['artg_training_math']}); (3) Training the Auto-Regressive Temporal Guidance (ARTG) module to leverage previously restored high-quality frames for improved temporal coherence with training objective (\ref{['tpm_training_math']}). Each module is trained separately before integrating them into the final framework.
• Figure 3: Overview of our pipeline. Given a low-quality (LQ) input frame, we first initialize its latent representation and employ an autoregressive diffusion model composed of a distilled denoising U-Net, autoregressive temporal Guidance, and an autoregressive temporal Decoder. Temporal guidance utilizes flow-warped high-quality (HQ) results from the previous frame to condition the current frame’s latent denoising and decoding processes, significantly improving perceptual quality and temporal consistency in an efficient, online manner.
• Figure 4: Qualitative comparison on REDS4 and Vimeo-90K-T datasets. Our method demonstrates superior visual quality with sharper details compared to unidirectional methods (TMP zhang2024tmp, RealViformer zhang2024realviformer) and competitive performance against bidirectional methods (StableVSR rota2024enhancing, MGLD-VSR yang2024motion, RVRT liang2022recurrent, BasicVSR++chan2022basicvsr++, RealBasicVSRchan2022investigating). Improvements include reduced artifacts and enhanced temporal stability (see zoomed patches).
• Figure 5: Ablation study on the Temporal Processor Module (TPM). Integrating TPM improves motion stability and reduces temporal artifacts by leveraging warped previous-frame features, enhancing temporal consistency in video super-resolution.