VipDiff: Towards Coherent and Diverse Video Inpainting via Training-free Denoising Diffusion Models

TL;DR

VipDiff addresses the challenge of temporal coherence in video inpainting without training a video diffusion model. It conditioning a frozen image-level diffusion model via optical-flow guided pixel propagation and a noise-optimized reverse diffusion objective, enabling both coherence and diversity across frames. The approach leverages flow completion (RAFT), backward pixel warping, and a constrained diffusion loss to fill masked regions while propagating valid pixels to neighboring frames. Empirical results on YouTube-VOS and DAVIS show state-of-the-art spatial-temporal fidelity with modest computation on a single GPU, demonstrating the practicality and scalability of training-free diffusion-based video inpainting.

Abstract

Recent video inpainting methods have achieved encouraging improvements by leveraging optical flow to guide pixel propagation from reference frames either in the image space or feature space. However, they would produce severe artifacts in the mask center when the masked area is too large and no pixel correspondences can be found for the center. Recently, diffusion models have demonstrated impressive performance in generating diverse and high-quality images, and have been exploited in a number of works for image inpainting. These methods, however, cannot be applied directly to videos to produce temporal-coherent inpainting results. In this paper, we propose a training-free framework, named VipDiff, for conditioning diffusion model on the reverse diffusion process to produce temporal-coherent inpainting results without requiring any training data or fine-tuning the pre-trained diffusion models. VipDiff takes optical flow as guidance to extract valid pixels from reference frames to serve as constraints in optimizing the randomly sampled Gaussian noise, and uses the generated results for further pixel propagation and conditional generation. VipDiff also allows for generating diverse video inpainting results over different sampled noise. Experiments demonstrate that VipDiff can largely outperform state-of-the-art video inpainting methods in terms of both spatial-temporal coherence and fidelity.

Figures (6)

• Figure 1: Video inpainting results on Davis dataset, we randomly select 3 frames from 'bear' and 'cows' videos. LDM generates different contents, ProPainter and ECFVI contain artifacts in the mask center, while our samples are more sharp and temporal-coherent.
• Figure 2: Overall framework of our VipDiff. Given a target frame $x^{k}_{0}$, we first adopt a flow completion model to predict the optical flows. Then the flows are utilized for pixel propagation to extract temporal prior from reference frames to get partially inpainted image $\widetilde{x}^{k}_{0}$. Next, $\widetilde{x}^{k}_{0}$ will act as constrains for optimizing the random sampled Gaussian nosie $z$ which is feed for the reverse denoising U-Net. Through backpropagation, we optimize the noise $z$ at each time step and finally find an optimal $z^{*}$ for filling the target frame. Note that all the model parameters are frozen during the noise optimization process.
• Figure 3: Process for propagating constrained generated pixels to other frames.
• Figure 4: Qualitative comparisons with SOTA video inpainting methods. Best viewed in PDF with zoom. Please refer to the supplementary video for a comprehensive comparison.
• Figure 5: Ablation study on different variants. From top row to the bottom, they are (a) input, (b) output of LDM, (c) LDM combined pixel propagation process, (d) our framework without reverse noise optimization, and (e) ours.