Small Clips, Big Gains: Learning Long-Range Refocused Temporal Information for Video Super-Resolution

TL;DR

The paper tackles the challenge of learning long-range temporal dependencies in video super-resolution (VSR) without prohibitive training costs. It introduces LRTI-VSR, a TBPTT-inspired framework that performs forward propagation on long sequences to learn temporal propagation while training on shorter clips, coupled with a Refocused Intra&Inter-Frame Transformer Block (RITB) that uses sparse ReLU^2 attention and a Refocused Gated Unit to selectively leverage useful temporal cues. Empirically, LRTI-VSR achieves state-of-the-art performance on long-video benchmarks (REDS4 and ToS3) with favorable efficiency and demonstrates strong gains over both CNN- and Transformer-based baselines. The work offers a practical approach to scalable long-range VSR and provides a foundation for future efficiency-driven transformer designs in video restoration.

Abstract

Video super-resolution (VSR) can achieve better performance compared to single image super-resolution by additionally leveraging temporal information. In particular, the recurrent-based VSR model exploits long-range temporal information during inference and achieves superior detail restoration. However, effectively learning these long-term dependencies within long videos remains a key challenge. To address this, we propose LRTI-VSR, a novel training framework for recurrent VSR that efficiently leverages Long-Range Refocused Temporal Information. Our framework includes a generic training strategy that utilizes temporal propagation features from long video clips while training on shorter video clips. Additionally, we introduce a refocused intra&inter-frame transformer block which allows the VSR model to selectively prioritize useful temporal information through its attention module while further improving inter-frame information utilization in the FFN module. We evaluate LRTI-VSR on both CNN and transformer-based VSR architectures, conducting extensive ablation studies to validate the contribution of each component. Experiments on long-video test sets demonstrate that LRTI-VSR achieves state-of-the-art performance while maintaining training and computational efficiency.

Figures (11)

• Figure 1: The pipline of our proposed training strategy. Our proposed training strategy can effectively utilize accurate long-range temporal dependencies in long video sequences to assist training while using short video clips for high training efficiency. In this figure, $\color{red}{\longrightarrow}$$h_{f}$ means forward propagation hidden state and $\color{blue}$$\color{blue}{\longleftarrow}$$h_{b}$ means backward paopagation hidden state in the bidirectional recurrent-based VSR model.
• Figure 2: Structure of feature propagation module. The commonly used feature propagation module contains bidirectional feature propagation structure (red solid lines), and second-order connection structure (green solid lines) that leverages the computed hidden states of the previous two frames to recover the current frame. These two components are widely used in existing state-of-the-art VSR models chan2022basicvsr++liang2022recurrentshi2022rethinkingxu2023implicitZhou_2024_CVPR.
• Figure 3: Attention map comparison. We compare the visualization of the intra&inter attention map under $\texttt{ReLU}^2$ and the original $\texttt{SoftMax}$. The Top 50 values of all tokens (64$\times$192) of an attention map under different network depths using these two activation functions are also counted. It is obvious that $\texttt{ReLU}^2$ is significantly more focused on fewer tokens than $\texttt{SoftMax}$.
• Figure 4: Illustration of the refocused intra&inter frame transformer block (RITB). The refocused intra&inter frame attention module contained in the RITB block can selectively prioritize useful temporal information. And the refocused gated unit module further facilitates the utilization of inter-frame information. More details of our RITB block can be found in Subsection \ref{['sec:ritb']}.
• Figure 5: Effect of truncated back-propagation (TB) training strategy. The contribution of our proposed training strategy is more obvious in regions with fine details. The long-term propagation information from long video clips leads to marked improvements.