Learnable Gated Temporal Shift Module for Deep Video Inpainting

TL;DR

This work tackles free-form video inpainting by enabling 2D CNNs to exploit temporal context through Learnable Gated Temporal Shift Module (LGTSM), which learns flexible temporal shifts and gating to handle masked regions. The approach combines a gated temporal-shift mechanism with a GAN-based loss (TSMGAN) and a multi-term objective including $L_{l_1}$, perceptual, and style losses to produce temporally coherent, high-quality video completions. Empirical results on FaceForensics and FVI show state-of-the-art performance with only 33% of the parameters and inference time of 3D convolution-based methods, and ablations demonstrate the critical roles of gating, temporal shifting, and the learnable kernels. The work advances efficient temporal modeling for video inpainting, with potential impact on video editing, restoration, and post-processing tasks where computational resources are limited.

Abstract

How to efficiently utilize temporal information to recover videos in a consistent way is the main issue for video inpainting problems. Conventional 2D CNNs have achieved good performance on image inpainting but often lead to temporally inconsistent results where frames will flicker when applied to videos (see https://www.youtube.com/watch?v=87Vh1HDBjD0&list=PLPoVtv-xp_dL5uckIzz1PKwNjg1yI0I94&index=1); 3D CNNs can capture temporal information but are computationally intensive and hard to train. In this paper, we present a novel component termed Learnable Gated Temporal Shift Module (LGTSM) for video inpainting models that could effectively tackle arbitrary video masks without additional parameters from 3D convolutions. LGTSM is designed to let 2D convolutions make use of neighboring frames more efficiently, which is crucial for video inpainting. Specifically, in each layer, LGTSM learns to shift some channels to its temporal neighbors so that 2D convolutions could be enhanced to handle temporal information. Meanwhile, a gated convolution is applied to the layer to identify the masked areas that are poisoning for conventional convolutions. On the FaceForensics and Free-form Video Inpainting (FVI) dataset, our model achieves state-of-the-art results with simply 33% of parameters and inference time.

Figures (4)

• Figure 1: Our model takes videos with free-form masks (first row) and fills in the missing areas with proposed LGTSM to generate realistic completed results (second row) compared to the original videos (third row). It could be applied to video editing tasks such as video object removal, as shown in the first two columns. Best viewed in color and zoom-in. See corresponding videos in the following links: https://www.youtube.com/watch?v=585xZjcmUlA&list=PLPoVtv-xp_dIc_qhYe5lrOyWCLEu7hWoL&index=4&t=0s, https://www.youtube.com/watch?v=08MNqZla29g&list=PLPoVtv-xp_dL5uckIzz1PKwNjg1yI0I94&index=37, and https://www.youtube.com/watch?v=3uiOOyimBHw&list=PLPoVtv-xp_dKLIZkMBvhhCu97AsXQ9e_D&index=2&t=0s.
• Figure 2: Explanation of the learnable shifting kernels in the proposed LGTSM. (a) Input features for the layer. We will do shifting operation on channel $\times$ time dimensions. (b) Original TSM from lin2018temporal. (c) Equivalent TSM by temporal shifting kernels. (d) In the proposed LGTSM, temporal shifting kernels are also learnable and the size could be different.
• Figure 3: Module design. We integrate (a) Residual TSM lin2018temporal and (b) gated convolution yu2018free to (c) Gated Temporal Shift Module (GTSM) and design learnable temporal shifting kernel (Fig. \ref{['fig:LTSM']}c) to make (d) the proposed Learnable Gated Temporal Shift Module (LGTSM).
• Figure 4: Visual comparison with the baselines on the FVI testing set with object-like masks. Best viewed in color and zoom-in. See https://www.youtube.com/watch?v=87Vh1HDBjD0&list=PLPoVtv-xp_dL5uckIzz1PKwNjg1yI0I94&index=32&t=0s.