TMP: Temporal Motion Propagation for Online Video Super-Resolution

TL;DR

This work proposes an efficient Temporal Motion Propagation (TMP) method, which leverages the continuity of motion field to achieve fast pixel-level alignment among consecutive frames and performs spatial-wise weighting on the warped features to enhance the robustness of alignment.

Abstract

Online video super-resolution (online-VSR) highly relies on an effective alignment module to aggregate temporal information, while the strict latency requirement makes accurate and efficient alignment very challenging. Though much progress has been achieved, most of the existing online-VSR methods estimate the motion fields of each frame separately to perform alignment, which is computationally redundant and ignores the fact that the motion fields of adjacent frames are correlated. In this work, we propose an efficient Temporal Motion Propagation (TMP) method, which leverages the continuity of motion field to achieve fast pixel-level alignment among consecutive frames. Specifically, we first propagate the offsets from previous frames to the current frame, and then refine them in the neighborhood, which significantly reduces the matching space and speeds up the offset estimation process. Furthermore, to enhance the robustness of alignment, we perform spatial-wise weighting on the warped features, where the positions with more precise offsets are assigned higher importance. Experiments on benchmark datasets demonstrate that the proposed TMP method achieves leading online-VSR accuracy as well as inference speed. The source code of TMP can be found at https://github.com/xtudbxk/TMP.

Figures (9)

• Figure 1: (a) Existing online-VSR methods compute the motion field of each frame separately via a weight-sharing offsets estimation module, while (b) our method propagates the motion field of previous frame to the current frame, reducing much the computational cost.
• Figure 2: Overview of our proposed online-VSR method. Left: The flowchart of the proposed method. There are two major differences between our method and the existing methods. One is the temporal motion propagation (TMP) module (highlighted in green color box), which propagates the motion field from the previous frame to the current frame. The other is the motion confidence weighted fusion (highlighted in orange color box), which weighs the warped features by the accuracy of estimated offsets. Right: The detailed architecture of the TMP module. Best viewed in color.
• Figure 3: Illustration of (a) object (OBJ) and (b) camera (CAM) motion propagation paths. The OBJ path aims to locate moving objects in the current frame, while the CAM path matches the static regions. The orange arrow represents the estimated motion from $I^{LR}_{t-2}$ to $I^{LR}_{t-1}$, which starts from the blue point and ends at the orange point. The red arrow indicates the temporally propagated motion. In the CAM path, the green point in $I^{LR}_{t}$ has the same position as the orange point in $I^{LR}_{t-1}$. The red points indicates the potential positions of the object at the corresponding frames, and the brighter colors represent higher likelihood.
• Figure 4: Qualitative comparisons on details of static regions and moving objects. The proposed method restores more details on both static regions (top two rows) and moving objects (bottom two rows) with CAM and OBJ motions, respectively. The results are from Vimeo-90K-T toflow. Zoom in for the best view.
• Figure 5: Qualitative results on REDS4 reds. Our TMP method recovers more details on the frames that contain various motions. Zoom-in for best view.