Enhancing Deformable Convolution based Video Frame Interpolation with Coarse-to-fine 3D CNN

TL;DR

Video frame interpolation remains challenging under large and complex motions. The authors propose a deformable-convolution based VFI method augmented with a coarse-to-fine 3D CNN to predict multi-flows $F_{tn}=(\alpha,\beta,\omega)$ for four input frames $I_0$–$I_3$, enabling accurate warping toward the intermediate time $t$ and occlusion-aware fusion. The approach combines 3D spatio-temporal feature extraction, per-frame context maps, and a GridNet-based synthesis network, trained with a Laplacian pyramid loss $L_{lap}$ and Charbonnier loss $L_{charb}$, yielding a total loss $L=L_{lap}+\lambda L_{charb}$. Experiments on Vimeo-90k, BVI-DVC, UCF101, DAVIS, and VFITex demonstrate state-of-the-art PSNR/SSIM with a PSNR gain up to $0.19$ dB, and ablation studies confirm the benefits of 3D filtering and the coarse-to-fine multi-flow refinement.

Abstract

This paper presents a new deformable convolution-based video frame interpolation (VFI) method, using a coarse to fine 3D CNN to enhance the multi-flow prediction. This model first extracts spatio-temporal features at multiple scales using a 3D CNN, and estimates multi-flows using these features in a coarse-to-fine manner. The estimated multi-flows are then used to warp the original input frames as well as context maps, and the warped results are fused by a synthesis network to produce the final output. This VFI approach has been fully evaluated against 12 state-of-the-art VFI methods on three commonly used test databases. The results evidently show the effectiveness of the proposed method, which offers superior interpolation performance over other state of the art algorithms, with PSNR gains up to 0.19dB.

Figures (3)

• Figure 1: Overall architecture of the proposed model.
• Figure 2: The architecture of the multi-flow block (MFB).
• Figure 3: Examples interpolated by best-performing VFI methods.