MaGGIe: Masked Guided Gradual Human Instance Matting

TL;DR

MaGGIe tackles multi-instance human matting in both images and videos by introducing a binary-mask guided framework that predicts per-instance alpha mattes in a single forward pass. The approach combines a mask-guided embedding, transformer-based coarse matte decoding, and sparse progressive refinement to maintain high detail with constant computational cost across many instances, supplemented by feature-level Conv-GRU and a temporal sparsity-based fusion for video. Key contributions include an efficient end-to-end architecture, temporal consistency at both feature and matte levels, and new image/video instance matting benchmarks and synthesis pipelines that bridge synthetic and real data. The work demonstrates strong image and video matting performance with robustness to mask quality and demonstrates practical impact for scalable, high-fidelity video cutouts in real-world pipelines.

Abstract

Human matting is a foundation task in image and video processing, where human foreground pixels are extracted from the input. Prior works either improve the accuracy by additional guidance or improve the temporal consistency of a single instance across frames. We propose a new framework MaGGIe, Masked Guided Gradual Human Instance Matting, which predicts alpha mattes progressively for each human instances while maintaining the computational cost, precision, and consistency. Our method leverages modern architectures, including transformer attention and sparse convolution, to output all instance mattes simultaneously without exploding memory and latency. Although keeping constant inference costs in the multiple-instance scenario, our framework achieves robust and versatile performance on our proposed synthesized benchmarks. With the higher quality image and video matting benchmarks, the novel multi-instance synthesis approach from publicly available sources is introduced to increase the generalization of models in real-world scenarios.

Figures (19)

• Figure 1: Our MaGGIe delivers precise and temporally consistent alpha mattes. It adeptly preserves intricate details and demonstrates robustness against noise in instance guidance masks by effectively utilizing information from adjacent frames. Red arrows highlight the areas of detailed zoom-in. (Optimally viewed in color and digital zoom in).
• Figure 2: Overall pipeline of MaGGIe. This framework processes frame sequences $\mathbf{I}$ and instance masks $\mathbf{M}$ to generate per-instance alpha mattes $\mathbf{A}'$ for each frame. It employs progressive refinement and sparse convolutions for accurate mattes in multi-instance scenarios, optimizing computational efficiency. The subfigures on the right illustrate the Instance Matte Decoder and the Instance Guidance, where we use mask guidance to predict coarse instance mattes and guide detail refinement by deep features, respectively. (Optimal in color and zoomed view).
• Figure 3: Variations of Masks for the Same Image in M-HIM2K Dataset. Masks generated using R50-C4-3x, R50-FPN-3x, R101-FPN-400e MaskRCNN models trained on COCO. (Optimal in color).
• Figure 4: Our model keeps steady memory and time complexity when the number of instance increases. InstMatt's complexity increases linearly with the number of instances.
• Figure 5: Enhanced Detail and Instance Separation by MaGGIe. Our model excels in rendering detailed outputs and effectively separating instances, as highlighted by red squares (detail focus) and red arrows (errors in other methods).