Referring Human Pose and Mask Estimation in the Wild

TL;DR

To enable prompt-conditioned estimation, this work proposes the first end-to-end promptable approach named UniPHD for R-HPM, which extracts multimodal representations and employs a proposed pose-centric hierarchical decoder to process (text or positional) instance queries and keypoint queries, producing results specific to the referred person.

Abstract

We introduce Referring Human Pose and Mask Estimation (R-HPM) in the wild, where either a text or positional prompt specifies the person of interest in an image. This new task holds significant potential for human-centric applications such as assistive robotics and sports analysis. In contrast to previous works, R-HPM (i) ensures high-quality, identity-aware results corresponding to the referred person, and (ii) simultaneously predicts human pose and mask for a comprehensive representation. To achieve this, we introduce a large-scale dataset named RefHuman, which substantially extends the MS COCO dataset with additional text and positional prompt annotations. RefHuman includes over 50,000 annotated instances in the wild, each equipped with keypoint, mask, and prompt annotations. To enable prompt-conditioned estimation, we propose the first end-to-end promptable approach named UniPHD for R-HPM. UniPHD extracts multimodal representations and employs a proposed pose-centric hierarchical decoder to process (text or positional) instance queries and keypoint queries, producing results specific to the referred person. Extensive experiments demonstrate that UniPHD produces quality results based on user-friendly prompts and achieves top-tier performance on RefHuman val and MS COCO val2017. Data and Code: https://github.com/bo-miao/RefHuman

Figures (5)

• Figure 1: Task illustration of (a) multi-person pose estimation predicts numerous outcomes and requires selection strategies during deployment, potentially leading to false negatives or suboptimal target results, and (b) our referring human pose and mask estimation requires a unified promptable model to simultaneously predict accurate pose and mask for the person of interest, providing comprehensive and identity-aware human representations to benefit human-AI interaction.
• Figure 2: Human-in-the-loop text prompt generation. We use GPT to generate descriptions with complementary local details and global context, then manually review/correct the descriptions.
• Figure 3: Detailed architecture of our UniPHD, which contains a multimodal encoder that imbues visual features with prompt awareness and a pose-centric hierarchical decoder that enables prompt-conditioned queries to effectively capture local details and global dependencies within targets. Our unified model is end-to-end and accepts text descriptions, scribbles, or points as prompts to predict the keypoint positions and segmentation mask of the target person.
• Figure 4: Qualitative results of our UniPHD with text prompts in various challenging scenarios.
• Figure 5: Qualitative results of our UniPHD with different prompts in various challenging scenarios.