VIP: Video Inpainting Pipeline for Real World Human Removal

TL;DR

VIP tackles real-world high-resolution video inpainting for removing humans without relying on text prompts. It builds on a state-of-the-art text-to-video diffusion backbone by introducing a motion module and progressive latent denoising via a Variational Autoencoder, coupled with a shadow-aware, KD-SAM-based segmentation of humans, belongings, and shadows. The framework also employs reference-image integration and a Dual-Fusion Latent Segment Refinement strategy to maintain temporal coherence across long sequences. Experimental results on YouTube-VOS-test and self-collected data show VIP achieves superior temporal consistency and visual fidelity, advancing practical capabilities for privacy, post-production, and effect-heavy workflows in real-world settings.

Abstract

Inpainting for real-world human and pedestrian removal in high-resolution video clips presents significant challenges, particularly in achieving high-quality outcomes, ensuring temporal consistency, and managing complex object interactions that involve humans, their belongings, and their shadows. In this paper, we introduce VIP (Video Inpainting Pipeline), a novel promptless video inpainting framework for real-world human removal applications. VIP enhances a state-of-the-art text-to-video model with a motion module and employs a Variational Autoencoder (VAE) for progressive denoising in the latent space. Additionally, we implement an efficient human-and-belongings segmentation for precise mask generation. Sufficient experimental results demonstrate that VIP achieves superior temporal consistency and visual fidelity across diverse real-world scenarios, surpassing state-of-the-art methods on challenging datasets. Our key contributions include the development of the VIP pipeline, a reference frame integration technique, and the Dual-Fusion Latent Segment Refinement method, all of which address the complexities of inpainting in long, high-resolution video sequences.

Figures (8)

• Figure 1: Video inpainting result generated by VIP, the comparison showcases show its ability to generate better inpainting result.
• Figure 2: The figure illustrates the training and inference processes for a Video Inpainting Unet model. In the training stage, we employ 3 parts as Input: Latent, Mask and Mask Latent. The bottom section depicts the inference pipeline. The inference process incorporates multiple stages of frame processing, including optical flow warpping and alignment, reference frame inpainting, and iterative inpainting steps before the final video inpainting unet. (For the sake of brevity, we omit the VAE encoding and deocde process in the inference pipeline.)
• Figure 3: Demonstration of our shadow detection and segmentation method) in comparison to the SAM2 image segmentation model). The red contours show the associated shadows detected and segmented by our algorithm. General-purpose segmentation models like SAM2 fail to accurately segment the corresponding shadows for the humans and lead to a bad generation. In contrast, our shadow detection and segmentation approach successfully segments and aligns shadows with the associated objects or humans, providing more precise and context-aware segmentation results which lead a perfect inpainting effect by SDXL inpainting model.
• Figure 4: Dual-Fusion Latent Segment Refinement Visualization.
• Figure 5: Qualitative comparisons on both video completion and object removal for high resolution videos.