AVID: Any-Length Video Inpainting with Diffusion Model

TL;DR

AVID tackles text-guided video inpainting for arbitrary-length videos by integrating motion modules into a pre-trained image inpainting diffusion model and introducing a structure guidance module to support diverse editing fidelities. It introduces Temporal MultiDiffusion with middle-frame attention guidance to enable zero-shot inference over variable video lengths, maintaining temporal coherence and identity consistency. The approach is validated on multiple inpainting types (object swap, re-texturing, uncropping) and durations, outperforming baselines in temporal consistency and background preservation, with quantitative metrics and user studies. This work enables practical, flexible video editing with minimal retraining, and suggests directions for stronger motion modeling and learnable structure control.

Abstract

Recent advances in diffusion models have successfully enabled text-guided image inpainting. While it seems straightforward to extend such editing capability into the video domain, there have been fewer works regarding text-guided video inpainting. Given a video, a masked region at its initial frame, and an editing prompt, it requires a model to do infilling at each frame following the editing guidance while keeping the out-of-mask region intact. There are three main challenges in text-guided video inpainting: ($i$) temporal consistency of the edited video, ($ii$) supporting different inpainting types at different structural fidelity levels, and ($iii$) dealing with variable video length. To address these challenges, we introduce Any-Length Video Inpainting with Diffusion Model, dubbed as AVID. At its core, our model is equipped with effective motion modules and adjustable structure guidance, for fixed-length video inpainting. Building on top of that, we propose a novel Temporal MultiDiffusion sampling pipeline with a middle-frame attention guidance mechanism, facilitating the generation of videos with any desired duration. Our comprehensive experiments show our model can robustly deal with various inpainting types at different video duration ranges, with high quality. More visualization results are made publicly available at https://zhang-zx.github.io/AVID/ .

Figures (16)

• Figure 1: Video inpainting. We introduce a video inpainting method versatile across a spectrum of video durations and tasks. Displayed frames are uniformly selected from videos of different lengths. The first row in the figure contains the source videos and the target regions, while the bottom row shows the results. The caption in the middle represents the language guidance and duration for each video.
• Figure 2: Overview of our method. In the training phase of our methodology, we employ a two-step approach. (a) Motion modules are integrated after each layer of the primary Text-to-Image (T2I) inpainting model, optimized for the video in-painting task via synthetic masks applied to the video data. (b) During the second training step, we fix the parameters in the UNet, $\epsilon_\theta$, and train a structure guidance module $\mathbf{s}_\theta$, leveraging a parameter copy from the UNet encoder. During inference, (c), for a video of length $N^\prime$, we construct a series of segments, each comprising $N$ successive frames. Throughout each denoising step, results for every segment are computed and aggregated.
• Figure 3: Middle-frame attention guidance. At inference, during each denoising step and within every self-attention layer, we retain the $K^{\lceil N^\prime / 2 \rceil}$ and $V^{\lceil N^\prime / 2 \rceil}$ values from the frame in the middle of the video. For the video's $i^{th}$ frame, we utilize its pixel queries, denoted as $Q^i$, to compute an auxiliary attention feature map. This is subsequently fused with the existing self-attention feature map within the same layer.
• Figure 4: Editing on videos of different durations. We employ our method on various videos and edit them for different tasks. We show the wide range of edits our approach can be used with different region sizes and video durations. Above each video, we note the number of frames, $N^\prime$, and the video duration.
• Figure 5: Comparison of various methods. We compare our method against several approaches, including per-frame inpainting using text-to-image LDM inpainting (PF) rombach2022high, Text2Video-Zero (T2V0) khachatryan2023text2video, VideoComposer (VC) wang2023videocomposer. All methods are evaluated using their default hyper-parameters as specified in either their corresponding publications or source codes. Each video in our experiments consists of $16$ frames. Our proposed approach successfully edits the videos as intended while retaining the details outside the designated target region. Moreover, our method upholds the editing capabilities of the image in-painting model we utilized. Notably, our results demonstrate remarkable consistency, outperforming other methods in our comparison.