Pose2Seg: Detection Free Human Instance Segmentation

TL;DR

Pose2Seg reframes human instance segmentation as a pose-guided task, arguing that human skeletal information provides clearer separation under heavy occlusion than bounding boxes. The authors introduce Affine-Align, a pose-based alignment mechanism, and Skeleton features (PAFs and part confidence maps) to guide a SegModule for instance masks, demonstrating improvements over detection-based methods, especially in occluded scenes. They also contribute OCHuman, a large occluded-human benchmark with bounding boxes, masks, and 17 joints, to stress-test occlusion handling. The results show substantial gains on occlusion scenarios and competitive performance on general cases, highlighting the practical impact of incorporating pose information into segmentation pipelines.

Abstract

The standard approach to image instance segmentation is to perform the object detection first, and then segment the object from the detection bounding-box. More recently, deep learning methods like Mask R-CNN perform them jointly. However, little research takes into account the uniqueness of the "human" category, which can be well defined by the pose skeleton. Moreover, the human pose skeleton can be used to better distinguish instances with heavy occlusion than using bounding-boxes. In this paper, we present a brand new pose-based instance segmentation framework for humans which separates instances based on human pose, rather than proposal region detection. We demonstrate that our pose-based framework can achieve better accuracy than the state-of-art detection-based approach on the human instance segmentation problem, and can moreover better handle occlusion. Furthermore, there are few public datasets containing many heavily occluded humans along with comprehensive annotations, which makes this a challenging problem seldom noticed by researchers. Therefore, in this paper we introduce a new benchmark "Occluded Human (OCHuman)", which focuses on occluded humans with comprehensive annotations including bounding-box, human pose and instance masks. This dataset contains 8110 detailed annotated human instances within 4731 images. With an average 0.67 MaxIoU for each person, OCHuman is the most complex and challenging dataset related to human instance segmentation. Through this dataset, we want to emphasize occlusion as a challenging problem for researchers to study.

Figures (8)

• Figure 1: Heavily occluded people are better separated using human pose than using bounding-box.
• Figure 2: Comparison of box-based alignment and our pose-based alignment (Affine-Align). Objects with strange pose are corrected to a standard pose.
• Figure 3: Samples of our OCHuman dataset. All the annotated people in this dataset are heavily occluded with others, and have comprehensive annotations.
• Figure 4: Overview of our network structure (Sec. \ref{['sec:overview']}). (a) Affine-Align operation (Sec. \ref{['sec:AffineAlign']}). (b) Skeleton features (Sec. \ref{['sec:Skel']}). (c) Structure of SegModule (Sec. \ref{['method:seg']}), in which residual unit refers to he2016deep.
• Figure 5: Pose templates clustered using K-means on COCO.