Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Video Inpainting with Aligned Frame Interval Guidance

Ming Xie, Junqiu Yu, Qiaole Dong, Xiangyang Xue, Yanwei Fu

TL;DR

This work tackles the instability and content degradation that plague chunk-based video inpainting. It introduces VidPivot, which decouples the task into multi-frame consistent image inpainting and masked area motion propagation, guided by frame interval priors. Key components include the FrameProp module for anchor-frame alignment with an ID-alignment adapter and a splicing mechanism, plus a context controller that injects contextual priors into a diffusion backbone to enforce temporal coherence. Extensive experiments on VPBench and DAVIS demonstrate competitive performance, strong identity preservation, and robust background consistency, with good generalization to editing tasks and challenging motion scenarios.

Abstract

Recent image-to-video (I2V) based video inpainting methods have made significant strides by leveraging single-image priors and modeling temporal consistency across masked frames. Nevertheless, these methods suffer from severe content degradation within video chunks. Furthermore, the absence of a robust frame alignment scheme compromises intra-chunk and inter-chunk spatiotemporal stability, resulting in insufficient control over the entire video. To address these limitations, we propose VidPivot, a novel framework that decouples video inpainting into two sub-tasks: multi-frame consistent image inpainting and masked area motion propagation. Our approach introduces frame interval priors as spatiotemporal cues to guide the inpainting process. To enhance cross-frame coherence, we design a FrameProp Module that implements a frame content propagation strategy, diffusing reference frame content into subsequent frames via a splicing mechanism. Additionally, a dedicated context controller encodes these coherent frame priors into the I2V generative backbone, effectively serving as soft constrain to suppress content distortion during generation. Extensive evaluations demonstrate that VidPivot achieves competitive performance across diverse benchmarks and generalizes well to different video inpainting scenarios.

Enhancing Video Inpainting with Aligned Frame Interval Guidance

TL;DR

This work tackles the instability and content degradation that plague chunk-based video inpainting. It introduces VidPivot, which decouples the task into multi-frame consistent image inpainting and masked area motion propagation, guided by frame interval priors. Key components include the FrameProp module for anchor-frame alignment with an ID-alignment adapter and a splicing mechanism, plus a context controller that injects contextual priors into a diffusion backbone to enforce temporal coherence. Extensive experiments on VPBench and DAVIS demonstrate competitive performance, strong identity preservation, and robust background consistency, with good generalization to editing tasks and challenging motion scenarios.

Abstract

Recent image-to-video (I2V) based video inpainting methods have made significant strides by leveraging single-image priors and modeling temporal consistency across masked frames. Nevertheless, these methods suffer from severe content degradation within video chunks. Furthermore, the absence of a robust frame alignment scheme compromises intra-chunk and inter-chunk spatiotemporal stability, resulting in insufficient control over the entire video. To address these limitations, we propose VidPivot, a novel framework that decouples video inpainting into two sub-tasks: multi-frame consistent image inpainting and masked area motion propagation. Our approach introduces frame interval priors as spatiotemporal cues to guide the inpainting process. To enhance cross-frame coherence, we design a FrameProp Module that implements a frame content propagation strategy, diffusing reference frame content into subsequent frames via a splicing mechanism. Additionally, a dedicated context controller encodes these coherent frame priors into the I2V generative backbone, effectively serving as soft constrain to suppress content distortion during generation. Extensive evaluations demonstrate that VidPivot achieves competitive performance across diverse benchmarks and generalizes well to different video inpainting scenarios.
Paper Structure (16 sections, 4 equations, 12 figures, 3 tables)

This paper contains 16 sections, 4 equations, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Multi-frame Consistent Inpainting
  5. Context Guidance via Controller Injection
  6. Masked Area Content Propagation
  7. Experiments
  8. Results of Video Inpainting
  9. Results of Video Editing
  10. Ablation Analysis
  11. Discussion about Challenging Scenarios
  12. Conclusion
  13. More qualitative results of VidPivot
  14. Discussion about Inference Time
  15. User Study
  16. ...and 1 more sections

Figures (12)

  • Figure 1: The inpainting results of our VidPivot. VidPivot can preserve consistent ID across inpainted videos, achieve coherent and stable masked area generation.
  • Figure 2: The pipeline of the FrameProp module. It generates both the I2V-masked video and the context video through three steps: Sampled Video Formation, Reference-guided Inpainting, and Spliced Video Formation.
  • Figure 3: The overall pipeline of VidPivot. As shown in the figure, VidPivot consists of three main components: the FrameProp module, the context controller, and the video diffusion transformer. The FrameProp module generates ID-aligned anchor frames, while the context controller extracts the upstream-constructed context video and injects it into the video diffusion transformer to guide temporal modeling.
  • Figure 4: The inpainting comparison between VidPivot, ProPainter, DiffuEraser and VideoPainter.
  • Figure 5: The frame-by-frame editing comparison between VidPivot, ProPainter and COCOCO.
  • ...and 7 more figures