PoseBH: Prototypical Multi-Dataset Training Beyond Human Pose Estimation

TL;DR

PoseBH tackles the core difficulties of multi-dataset training for pose estimation by introducing nonparametric keypoint prototypes learned in a unified embedding space and a cross-type self-supervision mechanism that aligns keypoint predictions with embeddings. This combination addresses skeletal heterogeneity and limited cross-dataset supervision, enabling robust generalization across humans, hands, and animals while preserving performance on standard benchmarks. The approach yields substantial gains over traditional MDT baselines, demonstrates strong domain transfer to InterHand2.6M and 3DPW, and maintains competitive results on COCO and MPII, highlighting its practical impact for broadpose estimation tasks. The work paves the way for more flexible, scalable MDT in pose estimation and suggests future directions toward zero-shot extension and 3D-domain adaptation.

Abstract

We study multi-dataset training (MDT) for pose estimation, where skeletal heterogeneity presents a unique challenge that existing methods have yet to address. In traditional domains, \eg regression and classification, MDT typically relies on dataset merging or multi-head supervision. However, the diversity of skeleton types and limited cross-dataset supervision complicate integration in pose estimation. To address these challenges, we introduce PoseBH, a new MDT framework that tackles keypoint heterogeneity and limited supervision through two key techniques. First, we propose nonparametric keypoint prototypes that learn within a unified embedding space, enabling seamless integration across skeleton types. Second, we develop a cross-type self-supervision mechanism that aligns keypoint predictions with keypoint embedding prototypes, providing supervision without relying on teacher-student models or additional augmentations. PoseBH substantially improves generalization across whole-body and animal pose datasets, including COCO-WholeBody, AP-10K, and APT-36K, while preserving performance on standard human pose benchmarks (COCO, MPII, and AIC). Furthermore, our learned keypoint embeddings transfer effectively to hand shape estimation (InterHand2.6M) and human body shape estimation (3DPW). The code for PoseBH is available at: https://github.com/uyoung-jeong/PoseBH.

Figures (12)

• Figure 1: PoseBH unifies diverse skeleton formats, including humans, hands, and animals. The displayed skeletons show pose estimation results from our method, with 3DPW and InterHand2.6M predictions from a transferred model.
• Figure 2: Overview of the PoseBH architecture. During training, the embedding head maps the backbone features into a unified keypoint embedding space. By matching these embeddings with prototypes, dataset-specific keypoint heatmaps $\mathbf{k}^{\text{P}}_{\text{coco}}$ are generated. Prototypes are updated nonparametrically from the embeddings. During inference, the embedding head and the subsequent procedures are removed.
• Figure 3: An illustrative example of cross-type self-supervision. Given a COCO image, we jointly refine the AIC head and AIC prototype predictions to produce reliable AIC heatmaps.
• Figure 4: Comparative pose estimation results on human (left) and animal (dog; right).
• Figure 5: Comparative results on InterHand2.6M (a--c) and 3DPW (e--f).