Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Masked 3D Diffusion Model for Video Outpainting

Fanda Fan, Chaoxu Guo, Litong Gong, Biao Wang, Tiezheng Ge, Yuning Jiang, Chunjie Luo, Jianfeng Zhan

TL;DR

Video outpainting must extend frame edges while maintaining temporal coherence across long sequences. We introduce a Masked 3D Diffusion Model (M3DDM) trained with bidirectional mask modeling and a hybrid coarse-to-fine pipeline that combines infilling (guided by the first/last frames) and interpolation (guided by intermediate frames), aided by cross-attention to $g$ global frames. The approach leverages latent diffusion models (LDMs) for memory efficiency and priors, and employs masked guide frames to reduce jitter and improve cross-clip consistency. Experiments show state-of-the-art results on video outpainting with reduced artifact accumulation, and the authors provide code at the project URL.

Abstract

Video outpainting aims to adequately complete missing areas at the edges of video frames. Compared to image outpainting, it presents an additional challenge as the model should maintain the temporal consistency of the filled area. In this paper, we introduce a masked 3D diffusion model for video outpainting. We use the technique of mask modeling to train the 3D diffusion model. This allows us to use multiple guide frames to connect the results of multiple video clip inferences, thus ensuring temporal consistency and reducing jitter between adjacent frames. Meanwhile, we extract the global frames of the video as prompts and guide the model to obtain information other than the current video clip using cross-attention. We also introduce a hybrid coarse-to-fine inference pipeline to alleviate the artifact accumulation problem. The existing coarse-to-fine pipeline only uses the infilling strategy, which brings degradation because the time interval of the sparse frames is too large. Our pipeline benefits from bidirectional learning of the mask modeling and thus can employ a hybrid strategy of infilling and interpolation when generating sparse frames. Experiments show that our method achieves state-of-the-art results in video outpainting tasks. More results and codes are provided at our https://fanfanda.github.io/M3DDM/.

Hierarchical Masked 3D Diffusion Model for Video Outpainting

TL;DR

Video outpainting must extend frame edges while maintaining temporal coherence across long sequences. We introduce a Masked 3D Diffusion Model (M3DDM) trained with bidirectional mask modeling and a hybrid coarse-to-fine pipeline that combines infilling (guided by the first/last frames) and interpolation (guided by intermediate frames), aided by cross-attention to global frames. The approach leverages latent diffusion models (LDMs) for memory efficiency and priors, and employs masked guide frames to reduce jitter and improve cross-clip consistency. Experiments show state-of-the-art results on video outpainting with reduced artifact accumulation, and the authors provide code at the project URL.

Abstract

Video outpainting aims to adequately complete missing areas at the edges of video frames. Compared to image outpainting, it presents an additional challenge as the model should maintain the temporal consistency of the filled area. In this paper, we introduce a masked 3D diffusion model for video outpainting. We use the technique of mask modeling to train the 3D diffusion model. This allows us to use multiple guide frames to connect the results of multiple video clip inferences, thus ensuring temporal consistency and reducing jitter between adjacent frames. Meanwhile, we extract the global frames of the video as prompts and guide the model to obtain information other than the current video clip using cross-attention. We also introduce a hybrid coarse-to-fine inference pipeline to alleviate the artifact accumulation problem. The existing coarse-to-fine pipeline only uses the infilling strategy, which brings degradation because the time interval of the sparse frames is too large. Our pipeline benefits from bidirectional learning of the mask modeling and thus can employ a hybrid strategy of infilling and interpolation when generating sparse frames. Experiments show that our method achieves state-of-the-art results in video outpainting tasks. More results and codes are provided at our https://fanfanda.github.io/M3DDM/.
Paper Structure (2 sections, 3 figures)

This paper contains 2 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work

Figures (3)

  • Figure 1: We propose a Masked 3D Diffusion Model (M3DDM) and a coarse-to-fine inference pipeline for video outpainting. Our method can not only generate high temporal consistency and reasonable outpainting results but also alleviate the problem of artifact accumulation in long video outpainting. The top row shows the first and last frames of five video clips. Each row below shows the video outpainting results of our method.
  • Figure 2: Artifact accumulation problem in long video outpainting. We compare two inference methods by our M3DDM: dense and coarse-to-fine (CTF) inferences. The index of the video frame is labeled above the image. This case shows horizontal video outpainting with a mask ratio of 0.5. We mark the area to be extended with a red line in the first image.
  • Figure 3: Masked 3D Diffusion Model Framework. During training, we concatenate corrupted raw video latents, random masked video latent, and masks before feeding them into the 3D UNet network. The network predicts the noise in the corrupted raw latents, allowing us to calculate the MSE loss with the added noise. Additionally, we uniformly select $g$ global frames from the video as a prompt and feed them into a trainable video encoder. Then the global frames feature map is placed in the cross-attention module of the 3D UNet.