Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

M3DDM+: An improved video outpainting by a modified masking strategy

Takuya Murakawa, Takumi Fukuzawa, Ning Ding, Toru Tamaki

TL;DR

M3DDM+ tackles a training–inference mismatch in M3DDM's masking strategy, where per-frame random masks during training cause temporal and spatial artifacts under information-scarce conditions. By enforcing uniform mask direction and width across all frames during training and fine-tuning the pretrained M3DDM, the method bridges the gap between training and inference. The approach improves visual fidelity and temporal coherence in challenging scenarios—such as static scenes or large outpainting regions—without increasing computational overhead. This makes high-quality video outpainting more robust for practical deployments with limited compute resources.</p>

Abstract

M3DDM provides a computationally efficient framework for video outpainting via latent diffusion modeling. However, it exhibits significant quality degradation -- manifested as spatial blur and temporal inconsistency -- under challenging scenarios characterized by limited camera motion or large outpainting regions, where inter-frame information is limited. We identify the cause as a training-inference mismatch in the masking strategy: M3DDM's training applies random mask directions and widths across frames, whereas inference requires consistent directional outpainting throughout the video. To address this, we propose M3DDM+, which applies uniform mask direction and width across all frames during training, followed by fine-tuning of the pretrained M3DDM model. Experiments demonstrate that M3DDM+ substantially improves visual fidelity and temporal coherence in information-limited scenarios while maintaining computational efficiency. The code is available at https://github.com/tamaki-lab/M3DDM-Plus.

M3DDM+: An improved video outpainting by a modified masking strategy

TL;DR

M3DDM+ tackles a training–inference mismatch in M3DDM's masking strategy, where per-frame random masks during training cause temporal and spatial artifacts under information-scarce conditions. By enforcing uniform mask direction and width across all frames during training and fine-tuning the pretrained M3DDM, the method bridges the gap between training and inference. The approach improves visual fidelity and temporal coherence in challenging scenarios—such as static scenes or large outpainting regions—without increasing computational overhead. This makes high-quality video outpainting more robust for practical deployments with limited compute resources.</p>

Abstract

M3DDM provides a computationally efficient framework for video outpainting via latent diffusion modeling. However, it exhibits significant quality degradation -- manifested as spatial blur and temporal inconsistency -- under challenging scenarios characterized by limited camera motion or large outpainting regions, where inter-frame information is limited. We identify the cause as a training-inference mismatch in the masking strategy: M3DDM's training applies random mask directions and widths across frames, whereas inference requires consistent directional outpainting throughout the video. To address this, we propose M3DDM+, which applies uniform mask direction and width across all frames during training, followed by fine-tuning of the pretrained M3DDM model. Experiments demonstrate that M3DDM+ substantially improves visual fidelity and temporal coherence in information-limited scenarios while maintaining computational efficiency. The code is available at https://github.com/tamaki-lab/M3DDM-Plus.
Paper Structure (16 sections, 4 figures, 1 table)

This paper contains 16 sections, 4 figures, 1 table.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. METHOD
  4. Diffusion Model
  5. Mask Strategy
  6. Training
  7. Inference
  8. EXPERIMENTAL RESULTS
  9. Experimental Setup
  10. Benchmark and baseline.
  11. Evaluation metrics.
  12. Implementation details.
  13. Results
  14. Quantitative results.
  15. Qualitative results.
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Comparison of M3DDM and our M3DDM+. Red boxes indicate the boundary between the input frame regions and the generated regions.
  • Figure 2: Failure examples of M3DDM generation. Red boxes indicate the boundary between the input frame regions and the generated regions. In this experiment, 25% or 66% of the lateral margins of a 16:9 input video are cropped, and video outpainting is performed to restore the original 16:9 aspect ratio. The left two columns present first frames of generated videos under a standard outpainting setting with camera motion, while the right two columns illustrate outpainting results for static videos constructed by replacing all frames of the original video with its first frame.
  • Figure 3: Comparison of mask strategies (left) and the training pipeline of M3DDM and M3DDM+ (right). In the comparison of mask strategy, the blue area indicates mask region and the white area indicates the input clip region. M3DDM applies frame-wise varying mask directions and ratios within each video sequence, whereas our method enforces spatiotemporally consistent masking across all frames. The 3D U-Net receives a channel-wise concatenation: noised latent representations of the input video, latent representations of the masked input, and a binary mask encoding the masked regions.
  • Figure 4: Qualitative comparison between M3DDM and M3DDM+. The red dotted line delineates the original video region (inside) from the outpainted region (outside). Our method effectively mitigates blur artifacts in challenging scenarios: static sequences with no camera motion (DAVIS Static) and cases requiring large mask ratios.