Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OSDEnhancer: Taming Real-World Space-Time Video Super-Resolution with One-Step Diffusion

Shuoyan Wei, Feng Li, Chen Zhou, Runmin Cong, Yao Zhao, Huihui Bai

TL;DR

OSDEnhancer tackles real-world STVSR by casting it as a one-step diffusion problem built on a pretrained diffusion backbone. It introduces a temporal-refinement and spatial-enhancement mixture-of-experts (TR-SE MoE) to separately learn temporal coherence and spatial detail, aided by a bidirectional deformable VAE decoder for robust cross-frame reconstruction. Starting from a linearly pre-interpolated latent, it achieves high-fidelity HR/HFR outputs with strong temporal consistency and robustness to complex degradations, while offering substantial efficiency gains over multi-step diffusion approaches. Empirical results on synthetic and real-world datasets show state-of-the-art performance across fidelity and perceptual metrics, with practical inference speedups (e.g., ~7x over certain diffusion-based STVSR baselines) enabling high-resolution video synthesis like a 97-frame $1024\times1024$ clip.

Abstract

Diffusion models (DMs) have demonstrated exceptional success in video super-resolution (VSR), showcasing a powerful capacity for generating fine-grained details. However, their potential for space-time video super-resolution (STVSR), which necessitates not only recovering realistic visual content from low-resolution to high-resolution but also improving the frame rate with coherent temporal dynamics, remains largely underexplored. Moreover, existing STVSR methods predominantly address spatiotemporal upsampling under simplified degradation assumptions, which often struggle in real-world scenarios with complex unknown degradations. Such a high demand for reconstruction fidelity and temporal consistency makes the development of a robust STVSR framework particularly non-trivial. To address these challenges, we propose OSDEnhancer, a novel framework that, to the best of our knowledge, represents the first method to achieve real-world STVSR through an efficient one-step diffusion process. OSDEnhancer initializes essential spatiotemporal structures through a linear pre-interpolation strategy and pivots on training temporal refinement and spatial enhancement mixture of experts (TR-SE MoE), which allows distinct expert pathways to progressively learn robust, specialized representations for temporal coherence and spatial detail, further collaboratively reinforcing each other during inference. A bidirectional deformable variational autoencoder (VAE) decoder is further introduced to perform recurrent spatiotemporal aggregation and propagation, enhancing cross-frame reconstruction fidelity. Experiments demonstrate that the proposed method achieves state-of-the-art performance while maintaining superior generalization capability in real-world scenarios.

OSDEnhancer: Taming Real-World Space-Time Video Super-Resolution with One-Step Diffusion

TL;DR

OSDEnhancer tackles real-world STVSR by casting it as a one-step diffusion problem built on a pretrained diffusion backbone. It introduces a temporal-refinement and spatial-enhancement mixture-of-experts (TR-SE MoE) to separately learn temporal coherence and spatial detail, aided by a bidirectional deformable VAE decoder for robust cross-frame reconstruction. Starting from a linearly pre-interpolated latent, it achieves high-fidelity HR/HFR outputs with strong temporal consistency and robustness to complex degradations, while offering substantial efficiency gains over multi-step diffusion approaches. Empirical results on synthetic and real-world datasets show state-of-the-art performance across fidelity and perceptual metrics, with practical inference speedups (e.g., ~7x over certain diffusion-based STVSR baselines) enabling high-resolution video synthesis like a 97-frame clip.

Abstract

Diffusion models (DMs) have demonstrated exceptional success in video super-resolution (VSR), showcasing a powerful capacity for generating fine-grained details. However, their potential for space-time video super-resolution (STVSR), which necessitates not only recovering realistic visual content from low-resolution to high-resolution but also improving the frame rate with coherent temporal dynamics, remains largely underexplored. Moreover, existing STVSR methods predominantly address spatiotemporal upsampling under simplified degradation assumptions, which often struggle in real-world scenarios with complex unknown degradations. Such a high demand for reconstruction fidelity and temporal consistency makes the development of a robust STVSR framework particularly non-trivial. To address these challenges, we propose OSDEnhancer, a novel framework that, to the best of our knowledge, represents the first method to achieve real-world STVSR through an efficient one-step diffusion process. OSDEnhancer initializes essential spatiotemporal structures through a linear pre-interpolation strategy and pivots on training temporal refinement and spatial enhancement mixture of experts (TR-SE MoE), which allows distinct expert pathways to progressively learn robust, specialized representations for temporal coherence and spatial detail, further collaboratively reinforcing each other during inference. A bidirectional deformable variational autoencoder (VAE) decoder is further introduced to perform recurrent spatiotemporal aggregation and propagation, enhancing cross-frame reconstruction fidelity. Experiments demonstrate that the proposed method achieves state-of-the-art performance while maintaining superior generalization capability in real-world scenarios.
Paper Structure (18 sections, 12 equations, 10 figures, 7 tables)

This paper contains 18 sections, 12 equations, 10 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Space-Time Video Super-Resolution (STVSR)
  4. Real-World Video Super-Resolution (VSR)
  5. Video Frame Interpolation (VFI)
  6. Method
  7. Preliminary
  8. Overall framework
  9. TR-SE MoE
  10. Bidirectional Deformable VAE Decoder
  11. Experiments
  12. Experimental Settings
  13. Comparison with State-of-the-Art Methods
  14. Ablation Study
  15. Conclusion
  16. ...and 3 more sections

Figures (10)

  • Figure 1: Performance and efficiency comparison on real-world STVSR. Our OSDEnhancer adopts a one-step diffusion framework with a temporal-refinement and spatial-enhancement mixture-of-experts (TR-SE MoE) architecture. OSDEnhancer demonstrates superior reconstruction on interpolated frames over the real-world VideoLQ dataset chan2022investigating, exhibiting clearer structures and details. Moreover, it achieves higher fidelity and comparable efficiency than state-of-the-art DM-based methods on the real-world MVSR4x dataset wang2023benchmark under generating a 97-frame $1024\times1024$ video with single-frame interpolation, while delivering a $\sim7\times$ speedup over the recent DM-based STVSR approach VEnhancer he2024venhancer on an NVIDIA A800 GPU. * DAM-VSR uses an extra DM of image super-resolution; only the main DM steps are reported.
  • Figure 2: The overall training pipeline of the proposed OSDEnhancer framework. Our method aims to generate an HR and HFR video with high fidelity and rich detail from a LR and LFR one via one-step diffusion. Specifically, we adopt the TR-SE MoE with progressive training to explicitly disentangle temporal and spatial degradations.
  • Figure 3: The illustration of the bidirectional deformable VAE decoder pipeline. Deformable recurrent blocks (DRBs) integrated into the upsampling layers of a 3D causal VAE decoder yang2024cogvideox, enabling multi-scale cross-frame compensation.
  • Figure 4: Qualitative comparison of interpolated frames on real-world videos from VideoLQ chan2022investigating. Left: overlay of adjacent LR frames.
  • Figure 5: Qualitative comparison of STVSR on the GoPro dataset nah2017deep with $4\times$ spatial upscaling and 7-frame interpolation (frames 2–8 are interpolated).
  • ...and 5 more figures