VSRM: A Robust Mamba-Based Framework for Video Super-Resolution

TL;DR

VSRM addresses the challenge of long-sequence video super-resolution by introducing a Mamba-based framework that achieves large receptive fields with linear complexity. The key innovations are the Dual Aggregation Mamba Block (DAMB), Deformable Cross-mamba Alignment (DCA), and Frequency Charbonnier-like Loss (FCL), which collectively enable robust spatio-temporal modeling, flexible motion compensation, and frequency-aware optimization. Empirical results on REDS and Vimeo-90K demonstrate state-of-the-art PSNR/SSIM improvements with favorable model efficiency, and ablations confirm the contribution of each component. The approach offers a solid backbone for VSR and holds promise for extending to other low-level video tasks demanding high-frequency detail preservation.

Abstract

Video super-resolution remains a major challenge in low-level vision tasks. To date, CNN- and Transformer-based methods have delivered impressive results. However, CNNs are limited by local receptive fields, while Transformers struggle with quadratic complexity, posing challenges for processing long sequences in VSR. Recently, Mamba has drawn attention for its long-sequence modeling, linear complexity, and large receptive fields. In this work, we propose VSRM, a novel \textbf{V}ideo \textbf{S}uper-\textbf{R}esolution framework that leverages the power of \textbf{M}amba. VSRM introduces Spatial-to-Temporal Mamba and Temporal-to-Spatial Mamba blocks to extract long-range spatio-temporal features and enhance receptive fields efficiently. To better align adjacent frames, we propose Deformable Cross-Mamba Alignment module. This module utilizes a deformable cross-mamba mechanism to make the compensation stage more dynamic and flexible, preventing feature distortions. Finally, we minimize the frequency domain gaps between reconstructed and ground-truth frames by proposing a simple yet effective Frequency Charbonnier-like loss that better preserves high-frequency content and enhances visual quality. Through extensive experiments, VSRM achieves state-of-the-art results on diverse benchmarks, establishing itself as a solid foundation for future research.

Figures (6)

• Figure 1: The network architecture of our method. a) Video Super-Resolution based on Mamba (VSRM). b) Feature Propagation block (FPB) consists of Deformable Cross-mamba Alignment (DCA) and Dual Aggregation Mamba Blocks (DAMB). c) Spatial-to-Temportal Mamba block (S2TMB). d) Temporal-to-Spatial Mamba block (T2SMB).
• Figure 2: Structure and scan directions of Mamba module. a) Spatial-to-Temporal Mamba (S2T-Mamba). b) Temporal-to-Spatial Mamba (T2S-Mamba). For simplicity, we exclude the initial normalization and the final residual.
• Figure 3: Temporal Gated Feed-forward Network (TGFN)
• Figure 4: Pipeline of our proposed Deformable Cross-mamba Alignment (DCA). We initialize a fixed reference region $r$ within the window $w$. This region is then adjusted to a dynamic reference region $\bar{r}$ using an offset $\epsilon_r$, which is learned from the features within $w$. Finally, DCA learns the affinity through the cross-mamba module to calculate the final result (aligned point).
• Figure 5: Qualitative comparisons on REDS4 and Vid4 datasets, VSRM achieves clearer and more precise results, revealing finer patterns.