Unified Long Video Inpainting and Outpainting via Overlapping High-Order Co-Denoising

TL;DR

The paper tackles long-form video editing for inpainting and outpainting with high controllability. It introduces a unified framework built on a pre-trained text-to-video diffusion model, integrating LoRA-based mask-conditioned fine-tuning and an overlapping high-order co-denoising pipeline to extend sequences without seams. Key contributions include a dual-region MSE loss for balanced hole filling and content preservation, a sliding-window Heun-solver-based denoising with Hamming blending, and extensive experiments showing improved PSNR/SSIM/LPIPS over baselines on long videos. The approach enables practical, scalable long-range editing with minimal overhead, offering a path toward more versatile long-form video applications.

Abstract

Generating long videos remains a fundamental challenge, and achieving high controllability in video inpainting and outpainting is particularly demanding. To address both of these challenges simultaneously and achieve controllable video inpainting and outpainting for long video clips, we introduce a novel and unified approach for long video inpainting and outpainting that extends text-to-video diffusion models to generate arbitrarily long, spatially edited videos with high fidelity. Our method leverages LoRA to efficiently fine-tune a large pre-trained video diffusion model like Alibaba's Wan 2.1 for masked region video synthesis, and employs an overlap-and-blend temporal co-denoising strategy with high-order solvers to maintain consistency across long sequences. In contrast to prior work that struggles with fixed-length clips or exhibits stitching artifacts, our system enables arbitrarily long video generation and editing without noticeable seams or drift. We validate our approach on challenging inpainting/outpainting tasks including editing or adding objects over hundreds of frames and demonstrate superior performance to baseline methods like Wan 2.1 model and VACE in terms of quality (PSNR/SSIM), and perceptual realism (LPIPS). Our method enables practical long-range video editing with minimal overhead, achieved a balance between parameter efficient and superior performance.

Figures (4)

• Figure 1: Showcase of our methods. we introduce a novel and unified approach for long video inpainting and outpainting that extends text-to-video diffusion models to generate arbitrarily long, spatially edited videos with high fidelity.
• Figure 2: Overview. We introduce a unified LoRA-based fine-tuning pipeline for both video inpainting and outpainting on our InpaintBench benchmark. During training, each clip is randomly masked with either (i) border masks, which zero out frame edges, or (ii) interior masks, which occlude central regions; a dual-region MSE loss then encourages accurate hole-filling while preserving unmasked content. At inference, we partition long sequences into overlapping windows and perform temporal co-denoising using a two-stage Heun sampler with Hamming-window weighted blending, yielding seamless, artifact-free long-video editing.
• Figure 3: Qualitative Results. We illustrate two representative cases: (1) Inpainting (top): replacing a surfer with a dog and a husky with a panda. Our approach yields anatomically plausible animals, consistent lighting and texture, and smooth frame‐to‐frame motion—whereas Wan 2.1, Wan 2.1-Fun and VACE exhibit shape distortions, color/style mismatches or temporal jitter. (2) Outpainting (bottom): extending the ocean waves and the balcony scene. Ours produces seamless wave patterns and coherent architectural details (door, railing, floor) with no visible seams or flicker, while competing methods suffer from boundary artifacts, drift or inconsistent motion.
• Figure 4: Ablation study on dual-region MSE loss weight. We study the impact of balancing masked-region versus unmasked-region supervision on inpainting (left) and outpainting (right). The top row shows the same masked input sequence, and each subsequent row presents reconstructions with $\lambda=0.1$, $0.5$, and $0.9$. At $\lambda=0.1$, the model under-fills masked areas—preserving context but leaving visible gaps; at $\lambda=0.5$, hole filling improves at the expense of mild distortion in unmasked regions; and at $\lambda=0.9$, we observe the best trade-off, with sharp, semantically accurate completions that faithfully preserve all unmasked content.