Information Prebuilt Recurrent Reconstruction Network for Video Super-Resolution

TL;DR

An end-to-end information prebuilt recurrent reconstruction network (IPRRN), consisting of an information prebuilt network (IPNet) and a recurrent reconstruction network (RRNet), which can effectively achieve better quantitative and qualitative evaluation performance compared to the existing state-of-the-art methods.

Abstract

The video super-resolution (VSR) method based on the recurrent convolutional network has strong temporal modeling capability for video sequences. However, the temporal receptive field of different recurrent units in the unidirectional recurrent network is unbalanced. Earlier reconstruction frames receive less spatio-temporal information, resulting in fuzziness or artifacts. Although the bidirectional recurrent network can alleviate this problem, it requires more memory space and fails to perform many tasks with low latency requirements. To solve the above problems, we propose an end-to-end information prebuilt recurrent reconstruction network (IPRRN), consisting of an information prebuilt network (IPNet) and a recurrent reconstruction network (RRNet). By integrating sufficient information from the front of the video to build the hidden state needed for the initially recurrent unit to help restore the earlier frames, the information prebuilt network balances the input information difference at different time steps. In addition, we demonstrate an efficient recurrent reconstruction network, which outperforms the existing unidirectional recurrent schemes in all aspects. Many experiments have verified the effectiveness of the network we propose, which can effectively achieve better quantitative and qualitative evaluation performance compared to the existing state-of-the-art methods.

Figures (9)

• Figure 1: The comparison of the restoration effect at different time steps on City of Vid4. RRNet is used here as a unidirectional recurrent network.
• Figure 2: The structure of IPRRN, which includes a IPNet and a RRNet. The IPNet generates the initial hidden state $h_0$ through the processing of the early m frames. The RRNet is an efficient unidirectional recurrent network, which recovers LR frame to corresponding SR frame by inputting two adjacent frames and the hidden state.
• Figure 3: Structure of proposed IPNet. First, early m frames were fed into a group convolution to extract shallow features, then the SE block and a 1$\times$1 convolution were used for feature screening, and then input the features into the deep feature extraction module. Finally, the propagation module was used to normalized the deep features into hidden state space.
• Figure 4: Structure of proposed RRNet. RRNet is based on the unidirectional recurrent convolutional network, and it uses RDBs to extract and fuse features sufficiently to reconstruct the reference frame.
• Figure 5: RRNet compared to IPNet+RRNet on Vid4. The PSNR of the initial frame is relatively low when IPNet does not exist. After adding IPNet, the performance of earlier frames has improved dramatically, and this improvement will be continued during the length of the temporal receptive field.