Motion-Aware Video Frame Interpolation

TL;DR

This work tackles the challenge of video frame interpolation under non-linear motion, occlusions, and blur by proposing MA-VFI, a Motion-Aware VFI network that directly predicts intermediate optical flow. It employs a hierarchical pyramid feature extractor to capture multi-scale context and a cross-scale motion structure to iteratively refine four-scale intermediate flow maps, with an intermediate-flow–directed loss guiding learning. The approach achieves strong performance across multiple public datasets while maintaining efficiency close to real-time, and ablations confirm the importance of warped features and the designed loss terms. Overall, MA-VFI presents a practical, scalable solution that improves frame synthesis quality without prohibitive computational cost, enabling better real-time VFI on devices with limited resources.

Abstract

Video frame interpolation methodologies endeavor to create novel frames betwixt extant ones, with the intent of augmenting the video's frame frequency. However, current methods are prone to image blurring and spurious artifacts in challenging scenarios involving occlusions and discontinuous motion. Moreover, they typically rely on optical flow estimation, which adds complexity to modeling and computational costs. To address these issues, we introduce a Motion-Aware Video Frame Interpolation (MA-VFI) network, which directly estimates intermediate optical flow from consecutive frames by introducing a novel hierarchical pyramid module. It not only extracts global semantic relationships and spatial details from input frames with different receptive fields, enabling the model to capture intricate motion patterns, but also effectively reduces the required computational cost and complexity. Subsequently, a cross-scale motion structure is presented to estimate and refine intermediate flow maps by the extracted features. This approach facilitates the interplay between input frame features and flow maps during the frame interpolation process and markedly heightens the precision of the intervening flow delineations. Finally, a discerningly fashioned loss centered around an intermediate flow is meticulously contrived, serving as a deft rudder to skillfully guide the prognostication of said intermediate flow, thereby substantially refining the precision of the intervening flow mappings. Experiments illustrate that MA-VFI surpasses several representative VFI methods across various datasets, and can enhance efficiency while maintaining commendable efficacy.

Figures (10)

• Figure 1: Running speed vs. PSNR values for different VFI methods, e.g. TOFlow xue2019video, DAIN bao2019depth, AdaCoF lee2020adacof, CAIN choi2020channel, SepConv niklaus2017video1, and proposed MA-VFI on UCF101 dataset. The magnitude of each circle signifies the quantity of model parameters it encompasses.
• Figure 2: Visual depiction of pixel motion amidst video frames. The blue circular indicators denote the pixel coordinates corresponding to neighboring video frames at temporal instance T=0 and T=1, while the tangerine markers embody the pixel coordinates of the intermediary frame computed under the presumption of linear motion. In contrast, the crimson markers encapsulate the pixel coordinates of the authentic intermediary frame at time $t$.
• Figure 3: An Outline of the Motion-Aware Video Frame Interpolation Network, referred to as MA-VFI. This architecture comprises two integral components: the Pyramid Features Module and the Cross-Scale Motion Structure. In the initial stage, low-level and high-level features need to be extracted from the input frames, which subsequently facilitate the computation of intermediate flow maps essential for interpolation.
• Figure 4: Overview of the cross-scale motion structure. It is a top-down structure with lateral connections. The structure computes intermediate flow maps at four scales and refines them with residual flows at each level.
• Figure 5: Sketch of intermediate flow block architecture. Note that $\oplus$ represents an element-wise multiplier.