End-to-End Human Instance Matting

TL;DR

A novel End-to-End Human Instance Matting (E2E-HIM) framework for simultaneous multiple instance matting in a more efficient manner and a large-scale human instance matting dataset comprising over 100,000 human images with instance alpha matte labels is constructed.

Abstract

Human instance matting aims to estimate an alpha matte for each human instance in an image, which is extremely challenging and has rarely been studied so far. Despite some efforts to use instance segmentation to generate a trimap for each instance and apply trimap-based matting methods, the resulting alpha mattes are often inaccurate due to inaccurate segmentation. In addition, this approach is computationally inefficient due to multiple executions of the matting method. To address these problems, this paper proposes a novel End-to-End Human Instance Matting (E2E-HIM) framework for simultaneous multiple instance matting in a more efficient manner. Specifically, a general perception network first extracts image features and decodes instance contexts into latent codes. Then, a united guidance network exploits spatial attention and semantics embedding to generate united semantics guidance, which encodes the locations and semantic correspondences of all instances. Finally, an instance matting network decodes the image features and united semantics guidance to predict all instance-level alpha mattes. In addition, we construct a large-scale human instance matting dataset (HIM-100K) comprising over 100,000 human images with instance alpha matte labels. Experiments on HIM-100K demonstrate the proposed E2E-HIM outperforms the existing methods on human instance matting with 50% lower errors and 5X faster speed (6 instances in a 640X640 image). Experiments on the PPM-100, RWP-636, and P3M datasets demonstrate that E2E-HIM also achieves competitive performance on traditional human matting.

Figures (8)

• Figure 1: Results of instance-level alpha mattes estimated by the proposed E2E-HIM and two state-of-the-art methods. (a) Input images. (b) Alpha mattes estimated by our E2E-HIM.(c) Alpha mattes estimated by ISSMatting instmattcrv (Mask R-CNN he2020mask + DIM xu2017deep). (d) Alpha mattes estimated by InstMatt sun2022instmatt (Mask R-CNN + FBAMatting forte2020fbamatting). (e) Trimaps used by ISSMatting and InstMatt, which are generated from the instance segmentation of Mask R-CNN.
• Figure 2: Architecture of the proposed E2E-HIM. The general perception network first extracts image features and instance latent codes. The united guidance network then utilizes the extracted image features and instance latent codes to generate the united semantics guidance. Finally, the instance matting network decodes the image features and united semantics guidance to estimate all instance-level alpha mattes.
• Figure 3: Structure of the united guidance network. A spatial attention module and a semantics embedding module are adopted to generate the guidance that identifies both the locations and semantic correspondences of the instances in the image.
• Figure 4: Visualization of the learned guidance. (a) Input image. (b) Context features from the General Perception Network. (c) Features of the first guidance head. (d) Features of the second guidance head.
• Figure 5: Examples of images in the proposed HIM-100K dataset. From top to bottom in the figure are the original images, instance segmentation annotations, instance bounding box annotations, and instance alpha matte annotations. The human instances in the images are marked in different colors.