Keypoints as Dynamic Centroids for Unified Human Pose and Segmentation

TL;DR

This work introduces Keypoints as Dynamic Centroid (KDC), a bottom-up, centroid-based framework that unifies human pose estimation with instance-level segmentation without requiring a box detector. It leverages KHDR disk-based keypoint heatmaps, KeyCentroid refinements, and MaskCentroid embeddings to cluster pixels into corresponding human instances even during rapid, occluded movements via a dynamic center of attraction in SegNet, culminating in the PoseSeg module that fuses pose and segmentation outputs. Through extensive evaluations on COCO, CrowdPose, and OCHuman, KDC achieves state-of-the-art or competitive keypoint and segmentation performance with real-time capabilities, supported by ablations on KHDR, KeyCentroid, dynamic MaskCentroids, and Gaussian optimizations. The approach significantly advances robust, scalable, and real-time unified human pose and segmentation, enabling improved analysis in crowded and dynamic environments.

Abstract

The dynamic movement of the human body presents a fundamental challenge for human pose estimation and body segmentation. State-of-the-art approaches primarily rely on combining keypoint heatmaps with segmentation masks but often struggle in scenarios involving overlapping joints or rapidly changing poses during instance-level segmentation. To address these limitations, we propose Keypoints as Dynamic Centroid (KDC), a new centroid-based representation for unified human pose estimation and instance-level segmentation. KDC adopts a bottom-up paradigm to generate keypoint heatmaps for both easily distinguishable and complex keypoints and improves keypoint detection and confidence scores by introducing KeyCentroids using a keypoint disk. It leverages high-confidence keypoints as dynamic centroids in the embedding space to generate MaskCentroids, allowing for swift clustering of pixels to specific human instances during rapid body movements in live environments. Our experimental evaluations on the CrowdPose, OCHuman, and COCO benchmarks demonstrate KDC's effectiveness and generalizability in challenging scenarios in terms of both accuracy and runtime performance. The implementation is available at: https://sites.google.com/view/niazahmad/projects/kdc.

Figures (10)

• Figure 1: PoseNet operation begins by generating keypoint heatmaps in the feature space using a disk representation $D_R$ to identify potential keypoint locations. It then introduces KeyCentroid to refine these keypoint coordinates to improve accuracy. SegNet leverages the KeyCentroid $K_c$ defined by PoseNet to establish MaskCentroid $M_c$, which is essential for clustering mask pixels corresponding to specific human instances.
• Figure 2: The overview of the proposed KDC model. PoseNet generates keypoint heatmaps and refines them with KeyCentroid $K_c$, improving keypoint accuracy. SegNet uses $K_c$ to create MaskCentroid $M_c$, clustering mask pixels for precise instance segmentation. The PoseSeg module integrates these outputs, resulting in accurate unified human pose estimation and instance-level segmentation.
• Figure 3: (a) presents Keypoint heatmap using keypoint disk, (b) shows Point-wise Gaussian optimization (PGO) where $\sigma$ values are defined for each keypoint (c) Indicates KeyCentroid defined for the right knee using the keypoint disk.
• Figure 4: (a) Introduces MaskCentroid a dynamic high confident keypoint; (b) presents a precise segmentation map; (c) indicates instance-level segmentation; and (d) shows unified representation of human pose and estimation.
• Figure 5: Visual results from various components of the system reveal initial mispredictions and inaccuracies in the keypoint heatmap (first row), corrected by KeyCentroid (second row). False pixel classification in segmentation with Static MaskCentroid (third row) was resolved using Dynamic MaskCentroid (fourth row). Unified human pose and segmentation are shown in the fifth row.