FMA-Net: Flow-Guided Dynamic Filtering and Iterative Feature Refinement with Multi-Attention for Joint Video Super-Resolution and Deblurring

TL;DR

Compared to conventional dynamic filtering, the FGDF enables the FMA-Net to effectively handle large motions into the VSRDB, and the stacked FRMA blocks trained with the novel temporal anchor (TA) loss, which temporally anchors and sharpens features, refine features in a coarse-to-fine manner through iterative updates.

Abstract

We present a joint learning scheme of video super-resolution and deblurring, called VSRDB, to restore clean high-resolution (HR) videos from blurry low-resolution (LR) ones. This joint restoration problem has drawn much less attention compared to single restoration problems. In this paper, we propose a novel flow-guided dynamic filtering (FGDF) and iterative feature refinement with multi-attention (FRMA), which constitutes our VSRDB framework, denoted as FMA-Net. Specifically, our proposed FGDF enables precise estimation of both spatio-temporally-variant degradation and restoration kernels that are aware of motion trajectories through sophisticated motion representation learning. Compared to conventional dynamic filtering, the FGDF enables the FMA-Net to effectively handle large motions into the VSRDB. Additionally, the stacked FRMA blocks trained with our novel temporal anchor (TA) loss, which temporally anchors and sharpens features, refine features in a course-to-fine manner through iterative updates. Extensive experiments demonstrate the superiority of the proposed FMA-Net over state-of-the-art methods in terms of both quantitative and qualitative quality. Codes and pre-trained models are available at: https://kaist-viclab.github.io/fmanet-site

Figures (15)

• Figure 1: Our FMA-Net outperforms state-of-the-art methods in both quantitative and qualitative results for $\times 4$ VSRDB.
• Figure 2: Comparison of $3 \times 3$ dynamic filtering. (a) conventional dynamic filtering at location $p$ with fixed surroundings and (b) our flow-guided dynamic filtering (FGDF, Sec. \ref{['FGDF']}) at position $p$ with variable surroundings guided by learned optical flow.
• Figure 3: The architecture of FMA-Net for video super-resolution and deblurring (VSRDB).
• Figure 4: (a) Structure of i$+1$-th FRMA block (Sec. \ref{['FRMA']}); (b) Structure of Multi-Attention. FFN refers to the feed-forward network of the transformer vaswani2017attentiondosovitskiy2020image.
• Figure 5: Visual results of different methods on REDS4 nah2019ntire, GoPro nah2017deep, and YouTube test sets. Best viewed in zoom.