Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploiting Optical Flow Guidance for Transformer-Based Video Inpainting

Kaidong Zhang, Jialun Peng, Jingjing Fu, Dong Liu

TL;DR

This paper tackles the problem of query degradation in transformer-based video inpainting by leveraging completed optical flows to guide attention and feature propagation. It introduces FGT++, comprising a Local Aggregation Flow Completion (LAFC) network and a Flow-Guided Transformer with Flow Guidance Feature Integration (FGFI) and Flow-Guided Feature Propagation (FGFP), plus temporally deformable MHSA (TD-MHSA) and a dual perspective MHSA (DP-MHSA). The method achieves superior quantitative and qualitative results on Youtube-VOS and DAVIS datasets, and ablations demonstrate the effectiveness of each component, including an amplitude loss in the Fourier domain to refine low-frequency content. The work advances practical video inpainting by combining accurate flow completion with flow-aware attention and efficient, windowed transformer design, achieving better spatiotemporal coherence while maintaining computational efficiency relative to fully global attention baselines.

Abstract

Transformers have been widely used for video processing owing to the multi-head self attention (MHSA) mechanism. However, the MHSA mechanism encounters an intrinsic difficulty for video inpainting, since the features associated with the corrupted regions are degraded and incur inaccurate self attention. This problem, termed query degradation, may be mitigated by first completing optical flows and then using the flows to guide the self attention, which was verified in our previous work - flow-guided transformer (FGT). We further exploit the flow guidance and propose FGT++ to pursue more effective and efficient video inpainting. First, we design a lightweight flow completion network by using local aggregation and edge loss. Second, to address the query degradation, we propose a flow guidance feature integration module, which uses the motion discrepancy to enhance the features, together with a flow-guided feature propagation module that warps the features according to the flows. Third, we decouple the transformer along the temporal and spatial dimensions, where flows are used to select the tokens through a temporally deformable MHSA mechanism, and global tokens are combined with the inner-window local tokens through a dual perspective MHSA mechanism. FGT++ is experimentally evaluated to be outperforming the existing video inpainting networks qualitatively and quantitatively.

Exploiting Optical Flow Guidance for Transformer-Based Video Inpainting

TL;DR

This paper tackles the problem of query degradation in transformer-based video inpainting by leveraging completed optical flows to guide attention and feature propagation. It introduces FGT++, comprising a Local Aggregation Flow Completion (LAFC) network and a Flow-Guided Transformer with Flow Guidance Feature Integration (FGFI) and Flow-Guided Feature Propagation (FGFP), plus temporally deformable MHSA (TD-MHSA) and a dual perspective MHSA (DP-MHSA). The method achieves superior quantitative and qualitative results on Youtube-VOS and DAVIS datasets, and ablations demonstrate the effectiveness of each component, including an amplitude loss in the Fourier domain to refine low-frequency content. The work advances practical video inpainting by combining accurate flow completion with flow-aware attention and efficient, windowed transformer design, achieving better spatiotemporal coherence while maintaining computational efficiency relative to fully global attention baselines.

Abstract

Transformers have been widely used for video processing owing to the multi-head self attention (MHSA) mechanism. However, the MHSA mechanism encounters an intrinsic difficulty for video inpainting, since the features associated with the corrupted regions are degraded and incur inaccurate self attention. This problem, termed query degradation, may be mitigated by first completing optical flows and then using the flows to guide the self attention, which was verified in our previous work - flow-guided transformer (FGT). We further exploit the flow guidance and propose FGT++ to pursue more effective and efficient video inpainting. First, we design a lightweight flow completion network by using local aggregation and edge loss. Second, to address the query degradation, we propose a flow guidance feature integration module, which uses the motion discrepancy to enhance the features, together with a flow-guided feature propagation module that warps the features according to the flows. Third, we decouple the transformer along the temporal and spatial dimensions, where flows are used to select the tokens through a temporally deformable MHSA mechanism, and global tokens are combined with the inner-window local tokens through a dual perspective MHSA mechanism. FGT++ is experimentally evaluated to be outperforming the existing video inpainting networks qualitatively and quantitatively.
Paper Structure (31 sections, 23 equations, 18 figures, 12 tables)

This paper contains 31 sections, 23 equations, 18 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Introduction
  3. Related Work
  4. Video Inpainting
  5. Image Inpainting
  6. Transformers in Computer Vision
  7. Method
  8. Flow Completion Network
  9. Local aggregation
  10. Edge loss
  11. Entire loss function
  12. Overview of Flow-Guided Transformer
  13. Proposed Solutions for Query Degradation
  14. Query degradation
  15. Flow guidance feature integration
  16. ...and 16 more sections

Figures (18)

  • Figure 1: Qualitative comparison between FGT zhang2022flow, E2FGVI liCvpr22vInpainting, and FGT++. FGT++ is capable of synthesizing more complete scene structure and finer details, which leads to better spatiotemporal coherence in video inpainting.
  • Figure 2: Our method consists of two steps. Firstly, we adopt the Local Aggregation Flow Completion (LAFC) network to complete the corrupted flows. Secondly, we synthesize the corrupted regions with the improved flow-guided transformer under the guidance of the completed optical flows. The "Flow-guided content propagation" module is optional. PEG: Position embedding generator.
  • Figure 3: Illustration of the proposed flow guidance feature integration (FGFI) module.
  • Figure 4: Procedure of the proposed flow-guided feature propagation (FGFP) module. All the local frames share the same FGFP block. We illustrate the details of one FGFP block between a single reference frame and the target frame for simplicity.
  • Figure 5: Comparison between F3N and FGF3N modules. SC and SS stand for the soft composition and soft split operations in FFM Liu_2021_FuseFormer, respectively. $\hat{F}$ denotes the completed optical flows.
  • ...and 13 more figures