MatAnyone: Stable Video Matting with Consistent Memory Propagation

TL;DR

MatAnyone tackles the challenges of target-assigned video matting by introducing a memory-based framework with region-adaptive memory propagation to maintain semantic stability in core areas while preserving boundary detail. The approach combines a consistent memory propagation module, object-level memory grouping, core-area supervision via segmentation data, and a recurrent refinement strategy for robust inference. It expands the training and evaluation foundation with VM800 and YouTubeMatte, and a training strategy that leverages large-scale segmentation data to improve generalization. Empirical results show state-of-the-art performance on synthetic and real benchmarks, demonstrating enhanced semantic stability and boundary fidelity for practical, real-world video matting tasks.

Abstract

Auxiliary-free human video matting methods, which rely solely on input frames, often struggle with complex or ambiguous backgrounds. To address this, we propose MatAnyone, a robust framework tailored for target-assigned video matting. Specifically, building on a memory-based paradigm, we introduce a consistent memory propagation module via region-adaptive memory fusion, which adaptively integrates memory from the previous frame. This ensures semantic stability in core regions while preserving fine-grained details along object boundaries. For robust training, we present a larger, high-quality, and diverse dataset for video matting. Additionally, we incorporate a novel training strategy that efficiently leverages large-scale segmentation data, boosting matting stability. With this new network design, dataset, and training strategy, MatAnyone delivers robust and accurate video matting results in diverse real-world scenarios, outperforming existing methods.

Figures (17)

• Figure 1: Our MatAnyone is capable of producing highly detailed and temporally consistent alpha mattes throughout a video. (a) It adapts to a variety of frame sizes and media types (e.g., films, games, smartphone videos), achieving fine-grained details at the image-matting level. (b) RVM lin2022rvm, an auxiliary-free video matting method, struggles with complex or ambiguous backgrounds. In contrast, our method effectively isolates the target object from such distractors, preserving a clean background and complete foreground parts. (c) Our method also excels at consistently tracking the target (i.e., the lady in pink) even in scenes containing multiple salient objects (i.e., the man and the lady). It accurately distinguishes between them even during their interactions. (Zoom-in for best view)
• Figure 2: Definitions and motivations for MatAnyone. (a) In a matting frame, the image can be broadly divided into two areas based on the alpha value: the core (semantic) and the boundary (fine-details). The core includes the background (alpha values of 0) and the solid foreground (alpha values of 1), while the boundary (highlighted in pink) encompasses areas with alpha values between 0 and 1. (b) Due to the under-defined setting, auxiliary-free methods like RVM lin2022rvm are easily confused by ambiguous background. Meanwhile, mask-guided methods like MaGGIe huynh2024maggie tend to break the segmentation prior they aim to leverage, due to the deficiency in video matting data.
• Figure 3: An overview of MatAnyone. MatAnyone is a memory-based framework for video matting. Given a target segmentation map in the first frame, our model achieves stable and high-quality matting through consistent memory propagation, with a region-adaptive memory fusion module to combine information from the previous and current frame. To overcome the scarcity of real video matting data, we incorporate a new training strategy that effectively leverages matting data for fine-grained matting details and segmentation data for semantic stability, with designed losses separately.
• Figure 4: Qualitative comparisons on real-world videos. Our MatAnyone significantly outperforms existing auxiliary-free (RVM lin2022rvm) and mask-guided (FTP-VM huang2023ftp and MaGGIe huynh2024maggie) approaches in both detail extraction and semantic accuracy. For the lowest row, while other methods all miss out on important body parts (i.e., head) and mistakenly take background pixels as foreground (due to similar colors), thus generating messy outputs, our method presents an accurate and visually clean output by even identifying the shadow near the boundary.
• Figure 5: Quantitative comparisons with MaGGIe huynh2024maggie on instance video matting. Despite MaGGIe using instance mask as guidance for each frame, our method shows better performance, achieving better stability in object tracking and finer alpha matte details.