SAM-pose2seg: Pose-Guided Human Instance Segmentation in Crowds

TL;DR

This work tackles occlusion-robust human instance segmentation by adapting the Segment Anything Model (SAM) 2.1 with minimal encoder changes and a pose-guided prompting strategy. It introduces PoseMaskRefine, a refinement scheme that leverages visible pose keypoints to guide iterative mask updates, and demonstrates that using the three most visible keypoints suffices for accurate segmentation. Evaluations across COCO, CIHP, and OCHuman show strong generalization, with competitive AP under detected and ground-truth poses, and ablations confirm the primary role of the first keypoint and the effectiveness of simple prompting in crowded scenes. The approach preserves SAM’s generalization while achieving occlusion-aware performance, offering a practical component for BMP loops and dataset annotation tasks.

Abstract

Segment Anything (SAM) provides an unprecedented foundation for human segmentation, but may struggle under occlusion, where keypoints may be partially or fully invisible. We adapt SAM 2.1 for pose-guided segmentation with minimal encoder modifications, retaining its strong generalization. Using a fine-tuning strategy called PoseMaskRefine, we incorporate pose keypoints with high visibility into the iterative correction process originally employed by SAM, yielding improved robustness and accuracy across multiple datasets. During inference, we simplify prompting by selecting only the three keypoints with the highest visibility. This strategy reduces sensitivity to common errors, such as missing body parts or misclassified clothing, and allows accurate mask prediction from as few as a single keypoint. Our results demonstrate that pose-guided fine-tuning of SAM enables effective, occlusion-aware human segmentation while preserving the generalization capabilities of the original model. The code and pretrained models will be available at https://mirapurkrabek.github.io/BBox-MaskPose.

Figures (12)

• Figure 1: Importance of correction points in the default SAM training method MaskRefine. Comparison on a small COCO + CIHP training subset of mean IoU of the ground-truth mask. The first iteration contains only one point, any other i-th iteration is prompted by i keypoints and a correction mask. It seems that the refinement mostly causes minor changes in the overall mask shape.
• Figure 2: Keypoint selection methods on SAM 2.1. Prompting methods MaxVis (full) and MaxSpread (dashed) on COCO and CIHP datasets. 6 keypoints is the best for both methods. MaxSpread outperforms MaxVis as shown in BMPv1.
• Figure 3: Keypoint selection methods for SAM 2.1 and SAM-pose2seg on COCO and CIHP datasets. SAM 2.1 with MaxSpread (thin line) peaks at 6 keypoints for both datasets. SAM-pose2seg with MaxSpread (dashed) and MaxVis (full) behaves the same on both datasets and peaks at 3 keypoints. We selected MaxVis method due to its simplicity. SAM-pose2seg outperforms SAM 2.1 with both selection methods on both datasets.
• Figure 4: Problematic bounding box usage in SAM 2 when multiple people are included. Note: a part of the other person’s hand is recognized incorrectly in both cases due to an incorrect pose keypoint.
• Figure 5: Inflated bounding box usage in base SAM 2.1 (middle) and SAM 1 (right). SAM 2.1 often incorporates unnecessary noise when the prompted bounding box does not fit the person exactly. In contrast, SAM 1 performance generally does not drop drastically.