UniHPR: Unified Human Pose Representation via Singular Value Contrastive Learning

TL;DR

UniHPR tackles the problem of learning a unified pose representation across RGB images, 2D keypoints, and 3D poses to enable versatile 2D/3D HPE. It introduces a singular value-based contrastive loss, termed Triplet-InfoNCE, that uses the largest eigenvalue of ${\mathcal{M}_x\mathcal{M}_x^T}$ to align three modalities in a shared space, complemented by standard pairwise contrastive learning. The architecture combines an HRNet image encoder, Transformer pose encoders, and a diffusion-based decoder to support both lifting-based and image-based 3D HPE within a single pipeline, achieving state-of-the-art results on Human3.6M and 3DPW under cross-domain evaluation and enabling effective pose retrieval and interpolation. The work demonstrates that joint cross-modal alignment yields robust, interpretable embeddings with practical impact for multi-modal pose generation and recognition tasks.

Abstract

In recent years, there has been a growing interest in developing effective alignment pipelines to generate unified representations from different modalities for multi-modal fusion and generation. As an important component of Human-Centric applications, Human Pose representations are critical in many downstream tasks, such as Human Pose Estimation, Action Recognition, Human-Computer Interaction, Object tracking, etc. Human Pose representations or embeddings can be extracted from images, 2D keypoints, 3D skeletons, mesh models, and lots of other modalities. Yet, there are limited instances where the correlation among all of those representations has been clearly researched using a contrastive paradigm. In this paper, we propose UniHPR, a unified Human Pose Representation learning pipeline, which aligns Human Pose embeddings from images, 2D and 3D human poses. To align more than two data representations at the same time, we propose a novel singular value-based contrastive learning loss, which better aligns different modalities and further boosts performance. To evaluate the effectiveness of the aligned representation, we choose 2D and 3D Human Pose Estimation (HPE) as our evaluation tasks. In our evaluation, with a simple 3D human pose decoder, UniHPR achieves remarkable performance metrics: MPJPE 49.9mm on the Human3.6M dataset and PA-MPJPE 51.6mm on the 3DPW dataset with cross-domain evaluation. Meanwhile, we are able to achieve 2D and 3D pose retrieval with our unified human pose representations in Human3.6M dataset, where the retrieval error is 9.24mm in MPJPE.

Figures (6)

• Figure 1: RGB image, 2D and 3D human pose embeddings extracted by corresponding encoders in the shared feature space. After conducting contrastive learning during the pre-training stage, the embeddings extracted from these three different data representations of the same training sample are close to each other and away from other negative samples.
• Figure 2: The training scheme of UniHPR. Steps 1 and 2 are representation learning stages, and Task-Specific Finetune is the finetuning stage for any specific task. During Step 1, the 2D and 3D pose embedding alignment is trained first with $\mathcal{L}_{pair}$, and in Step 2, the image encoder is aligned with frozen 2D and 3D pose encoders via $\mathcal{L}_{pair}$ and $\mathcal{L}_{triplet}$. In the Task-Specific Finetuning stage, encoders and decoders are trained jointly via both contrastive learning, $\mathcal{L}_{pair}$ and $\mathcal{L}_{triplet}$, and task loss.
• Figure 3: $\mathcal{L}_{pair}$ is applied three times for contrastive learning and the singular value based $\mathcal{L}_{triplet}$ focuses on aligning three representations at the same time.
• Figure 4: Cosine similarities between different data representations. The yellow line is the one trained only with three pair-wise losses, $\mathcal{L}_{pair}$, and the purple line is the training curve with additional singular value-based InfoNCE loss, $\mathcal{L}_{triplet}$. Our proposed singular value-based InfoNCE loss helps align the feature space.
• Figure 5: Interpolation of 3D human pose representations in Human3.6M dataset.