Video Super-Resolution Transformer

TL;DR

The paper addresses video super-resolution by adapting Transformer architecture to exploit local spatiotemporal structure and cross-frame feature alignment. It introduces the VSR-Transformer, featuring a spatial-temporal convolutional self-attention (STCSA) layer to capture locality and a bidirectional optical flow-based feed-forward (BOFF) to enable feature propagation and alignment across frames. The approach achieves state-of-the-art PSNR/SSIM on REDS4 and Vimeo-90K-T, with ablations confirming the critical roles of STCSA and BOFF. This work advances parallelizable, high-quality VSR by integrating locality-aware attention with flow-guided cross-frame interaction, offering practical benefits for video enhancement tasks.

Abstract

Video super-resolution (VSR), with the aim to restore a high-resolution video from its corresponding low-resolution version, is a spatial-temporal sequence prediction problem. Recently, Transformer has been gaining popularity due to its parallel computing ability for sequence-to-sequence modeling. Thus, it seems to be straightforward to apply the vision Transformer to solve VSR. However, the typical block design of Transformer with a fully connected self-attention layer and a token-wise feed-forward layer does not fit well for VSR due to the following two reasons. First, the fully connected self-attention layer neglects to exploit the data locality because this layer relies on linear layers to compute attention maps. Second, the token-wise feed-forward layer lacks the feature alignment which is important for VSR since this layer independently processes each of the input token embeddings without any interaction among them. In this paper, we make the first attempt to adapt Transformer for VSR. Specifically, to tackle the first issue, we present a spatial-temporal convolutional self-attention layer with a theoretical understanding to exploit the locality information. For the second issue, we design a bidirectional optical flow-based feed-forward layer to discover the correlations across different video frames and also align features. Extensive experiments on several benchmark datasets demonstrate the effectiveness of our proposed method. The code will be available at https://github.com/caojiezhang/VSR-Transformer.

Figures (10)

• Figure 1: The framework of video super-resolution Transformer. Given a low-resolution (LR) video, we first use an extractor to capture features of the LR videos. Then, a spatial-temporal convolutional self-attention and an optical flow-based feed-forward network model a sequence of continuous representations. Note that these two layers both have skip connections. Last, the reconstruction network restores a high-resolution video from the representations and the upsampling frames.
• Figure 2: Illustration of the spatial-temporal convolutional self-attention. The unfold operation is to extract sliding local patches from a batched input feature map, while the fold operation is to combine an array of sliding local patches into a large feature map.
• Figure 3: Illustration of the bidirectional optical flow-based feed-forward layer. Given a video sequence, we first bidirectionally estimate the forward and backward optical flows and wrap the feature maps with the responding optical flows. Then we learn a forward and backward propagation network to produce two sequences of features from concatenated wrapped features and LR frames. Last, we fusion these two feature sequences into one feature sequence.
• Figure 4: Qualitative comparison on the REDS4 dataset for $4\times$ VSR. Zoom in for the best view.
• Figure 5: Qualitative comparison on Vimeo-90K-T for 4× VSR. Zoom in for the best view.

Theorems & Definitions (10)

• Definition 1
• Definition 2
• Theorem 1
• Theorem 2
• Definition 3
• Definition 4
• Theorem 3
• Lemma 1
• Lemma 2
• Lemma 3