VideoMaMa: Mask-Guided Video Matting via Generative Prior

TL;DR

VideoMaMa leverages diffusion priors to convert binary segmentation masks into high-quality alpha mattes for videos, achieving strong zero-shot generalization to real footage despite synthetic training data. A two-stage training regime and semantic knowledge injection enable both detailed spatial matting and temporal coherence, while mask augmentation prevents trivial copying of inputs. The MA-V dataset, created by pseudo-labeling SA-V masks with VideoMaMa, provides over 50K real-world videos and enables substantial improvements when fine-tuning SAM2 to SAM2-Matte, yielding robust in-the-wild matting performance. Together, VideoMaMa and MA-V demonstrate a scalable path to high-quality video matting and set the stage for broader data-driven matting research.

Abstract

Generalizing video matting models to real-world videos remains a significant challenge due to the scarcity of labeled data. To address this, we present Video Mask-to-Matte Model (VideoMaMa) that converts coarse segmentation masks into pixel accurate alpha mattes, by leveraging pretrained video diffusion models. VideoMaMa demonstrates strong zero-shot generalization to real-world footage, even though it is trained solely on synthetic data. Building on this capability, we develop a scalable pseudo-labeling pipeline for large-scale video matting and construct the Matting Anything in Video (MA-V) dataset, which offers high-quality matting annotations for more than 50K real-world videos spanning diverse scenes and motions. To validate the effectiveness of this dataset, we fine-tune the SAM2 model on MA-V to obtain SAM2-Matte, which outperforms the same model trained on existing matting datasets in terms of robustness on in-the-wild videos. These findings emphasize the importance of large-scale pseudo-labeled video matting and showcase how generative priors and accessible segmentation cues can drive scalable progress in video matting research.

Figures (8)

• Figure 1: We introduce Video Mask-to-Matte Model (VideoMaMa), a diffusion-based model that generates high-quality alpha mattes from input binary segmentation masks obtained either from existing models such as SAM2 sam2 or from ground-truth segmentation masks in existing datasets such as SA-V sam2. Examples shown highlights our VideoMaMa's ability to capture fine-grained details including motion blur, and intricate boundary structures on natural video footage.
• Figure 2: Overview of VideoMaMa architecture. RGB frames and guide masks are processed through video diffusion U-Net layers to generate high-quality video mattes. Semantic injection with DINO features is applied during training.
• Figure 3: Examples of mask augmentation methods. Polygon and Downsampling degradation are applied at weak and strong augmentation levels.
• Figure 4: Qualitative examples from our MA-V dataset. We show RGB frames with our high-quality MA-V annotations and original SA-V sam2 masks for comparison. MA-V provides refined alpha mattes for diverse scenarios.
• Figure 5: Qualitative comparison on in-the-wild videos. We evaluate two settings: (1) all-frame mask-guided video matting where VideoMaMa is compared against MaGGIe maggie, and (2) first-frame mask-guided matting where SAM2-Matte is compared against MatAnyone matanyone. All methods use SAM2 sam2 to generate mask inputs.