Single-stage Multi-human Parsing via Point Sets and Center-based Offsets

TL;DR

The paper tackles instance-aware multi-human parsing (IAMHP), a challenging task requiring per-pixel part semantics and per-instance labeling. It introduces SMP, a single-stage framework that represents humans with point sets formed by body barycenters and part barycenters and uses center-based offsets to map parts to bodies, enabling parallel prediction without explicit grouping. Key innovations include the Refined Feature Retain (RFR) module for mask-attended feature refinement and the Mask of Interest Reclassify (MIR) module to refine classification via ROI-aligned features. On the MHPv2.0 benchmark, SMP achieves state-of-the-art AP$^p_{50}$, AP$^p_{vol}$, and PCP$50$ while offering faster training and inference, and it shows generalized performance on DensePose COCO as well.

Abstract

This work studies the multi-human parsing problem. Existing methods, either following top-down or bottom-up two-stage paradigms, usually involve expensive computational costs. We instead present a high-performance Single-stage Multi-human Parsing (SMP) deep architecture that decouples the multi-human parsing problem into two fine-grained sub-problems, i.e., locating the human body and parts. SMP leverages the point features in the barycenter positions to obtain their segmentation and then generates a series of offsets from the barycenter of the human body to the barycenters of parts, thus performing human body and parts matching without the grouping process. Within the SMP architecture, we propose a Refined Feature Retain module to extract the global feature of instances through generated mask attention and a Mask of Interest Reclassify module as a trainable plug-in module to refine the classification results with the predicted segmentation. Extensive experiments on the MHPv2.0 dataset demonstrate the best effectiveness and efficiency of the proposed method, surpassing the state-of-the-art method by 2.1% in AP50p, 1.0% in APvolp, and 1.2% in PCP50. In particular, the proposed method requires fewer training epochs and a less complex model architecture. We will release our source codes, pretrained models, and online demos to facilitate further studies.

Figures (5)

• Figure 1: The differences between top-down, bottom-up, and single-stage methods in multi-human parsing task. The single-stage method is parallel computing, and neither the instance detection of the top-down method nor the grouping process of the bottom-up method are required. Best viewed in color.
• Figure 2: The Illustration of our Single-stage Multi-human Parsing (SMP) framework for instance-aware human semantic parsing. Coord* refers to concatenating the relative coordinates to the feature. Details can be seen in Sec. 3.1 and Sec. 4.1. Best viewed in color.
• Figure 3: The detailed architecture about RFR module. It is placed after downsampling of category and kernel branch as shown in Fig. \ref{['Overview']}.
• Figure 4: The visualization results of intermediate outputs for SMP. Center preds and Category preds (Face) are the visual confidence maps (“preds” is short for “predictions”). Center output is generated by maxpooling layer based on center preds. Yellow crosses in offsets output are the part instance locations pointed by offsets and white crosses are human centers. (1), (2) denote different human instances. Except original image, parts mask and instance visualization, other visualizations are resized to $S \times S$ (S=40 during inference) as description in Sec. 3.1 and Sec. 4.1. Best viewed in color.
• Figure 5: The visualization results from SMP including instances with different scales and quantity. In the highlight row, the left is RP-RCNN, the right is SMP. Best viewed in color.