GlobalPaint: Spatiotemporal Coherent Video Outpainting with Global Feature Guidance

TL;DR

GlobalPaint tackles the challenge of spatiotemporally coherent video outpainting under limited temporal context by combining a hierarchical key-frame strategy with a diffusion-based backbone enhanced by 3D windowed attention and global feature guidance. The method introduces EST modules to expand joint spatiotemporal receptive fields and distills global cues from observed regions via OpenCLIP tokens, injected through cross-attention during denoising. A key-frame first pipeline is followed by an interpolation model to fill in intermediate frames, mitigating error accumulation. Experimental results on DAVIS and YouTube-VOS show improved reconstruction quality, more natural motion (lower FVD), and competitive perceptual metrics, with favorable computational efficiency compared to prior methods. This approach advances practical video outpainting by enabling robust global coherence across long sequences while maintaining efficiency.

Abstract

Video outpainting extends a video beyond its original boundaries by synthesizing missing border content. Compared with image outpainting, it requires not only per-frame spatial plausibility but also long-range temporal coherence, especially when outpainted content becomes visible across time under camera or object motion. We propose GlobalPaint, a diffusion-based framework for spatiotemporal coherent video outpainting. Our approach adopts a hierarchical pipeline that first outpaints key frames and then completes intermediate frames via an interpolation model conditioned on the completed boundaries, reducing error accumulation in sequential processing. At the model level, we augment a pretrained image inpainting backbone with (i) an Enhanced Spatial-Temporal module featuring 3D windowed attention for stronger spatiotemporal interaction, and (ii) global feature guidance that distills OpenCLIP features from observed regions across all frames into compact global tokens using a dedicated extractor. Comprehensive evaluations on benchmark datasets demonstrate improved reconstruction quality and more natural motion compared to prior methods. Our demo page is https://yuemingpan.github.io/GlobalPaint/

Figures (9)

• Figure 1: GlobalPaint is a diffusion-based video outpainting framework equipped with enhanced spatiotemporal modules and guided by global features. It generates spatiotemporally coherent results under various mask patterns.
• Figure 2: Overview of GlobalPaint. We follow the LDM inpainting formulation LDM by concatenating the noisy latent $z_t$, masked latent $z_m$, and downsampled mask $m_{\text{down}}$ as inputs to an inflated UNet. Global cues are extracted from the observed regions of all frames using a frozen OpenCLIP encoder openclip, distilled into compact global tokens by a dedicated Global Feature Extractor, and injected into the UNet to guide generation process. EST+G denotes the Enhanced Spatial-Temporal module with Global Feature guidance.
• Figure 3: Qualitative comparison with M3DDM M3DDM and MOTIA MOTIA.
• Figure 4: Ablation on EST module and global feature guidance.
• Figure 5: Ablation comparing hierarchical processing and sequential clip-by-clip processing.