Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LangPose: Language-Aligned Motion for Robust 3D Human Pose Estimation

Longyun Liao, Rong Zheng

TL;DR

LangPose tackles the ill-posed 2D-to-3D human pose lifting by injecting semantic action knowledge through text–motion alignment. It introduces a two-stage regime with semantic-alignment pretraining on a synthetic, fine-grained action dataset and subsequent fine-tuning on real 3D pose data without action labels, augmented by masked body-part and masked time-window strategies. The architecture uses dual BERT-style encoders for text and pose and a cross-modal contrastive objective, achieving state-of-the-art MPJPE/PCK on Human3.6M and MPI-INF-3DHP while maintaining lower inference cost than diffusion-based methods. This work demonstrates that incorporating semantic action information can robustify 3D HPE under occlusion and dynamic motion, with potential extensions to motion prediction and captioning.

Abstract

2D-to-3D human pose lifting is an ill-posed problem due to depth ambiguity and occlusion. Existing methods relying on spatial and temporal consistency alone are insufficient to resolve these problems especially in the presence of significant occlusions or high dynamic actions. Semantic information, however, offers a complementary signal that can help disambiguate such cases. To this end, we propose LangPose, a framework that leverages action knowledge by aligning motion embeddings with text embeddings of fine-grained action labels. LangPose operates in two stages: pretraining and fine-tuning. In the pretraining stage, the model simultaneously learns to recognize actions and reconstruct 3D poses from masked and noisy 2D poses. During the fine-tuning stage, the model is further refined using real-world 3D human pose estimation datasets without action labels. Additionally, our framework incorporates masked body parts and masked time windows in motion modeling, encouraging the model to leverage semantic information when spatial and temporal consistency is unreliable. Experiments demonstrate the effectiveness of LangPose, achieving SOTA level performance in 3D pose estimation on public datasets, including Human3.6M and MPI-INF-3DHP. Specifically, LangPose achieves an MPJPE of 36.7mm on Human3.6M with detected 2D poses as input and 15.5mm on MPI-INF-3DHP with ground-truth 2D poses as input.

LangPose: Language-Aligned Motion for Robust 3D Human Pose Estimation

TL;DR

LangPose tackles the ill-posed 2D-to-3D human pose lifting by injecting semantic action knowledge through text–motion alignment. It introduces a two-stage regime with semantic-alignment pretraining on a synthetic, fine-grained action dataset and subsequent fine-tuning on real 3D pose data without action labels, augmented by masked body-part and masked time-window strategies. The architecture uses dual BERT-style encoders for text and pose and a cross-modal contrastive objective, achieving state-of-the-art MPJPE/PCK on Human3.6M and MPI-INF-3DHP while maintaining lower inference cost than diffusion-based methods. This work demonstrates that incorporating semantic action information can robustify 3D HPE under occlusion and dynamic motion, with potential extensions to motion prediction and captioning.

Abstract

2D-to-3D human pose lifting is an ill-posed problem due to depth ambiguity and occlusion. Existing methods relying on spatial and temporal consistency alone are insufficient to resolve these problems especially in the presence of significant occlusions or high dynamic actions. Semantic information, however, offers a complementary signal that can help disambiguate such cases. To this end, we propose LangPose, a framework that leverages action knowledge by aligning motion embeddings with text embeddings of fine-grained action labels. LangPose operates in two stages: pretraining and fine-tuning. In the pretraining stage, the model simultaneously learns to recognize actions and reconstruct 3D poses from masked and noisy 2D poses. During the fine-tuning stage, the model is further refined using real-world 3D human pose estimation datasets without action labels. Additionally, our framework incorporates masked body parts and masked time windows in motion modeling, encouraging the model to leverage semantic information when spatial and temporal consistency is unreliable. Experiments demonstrate the effectiveness of LangPose, achieving SOTA level performance in 3D pose estimation on public datasets, including Human3.6M and MPI-INF-3DHP. Specifically, LangPose achieves an MPJPE of 36.7mm on Human3.6M with detected 2D poses as input and 15.5mm on MPI-INF-3DHP with ground-truth 2D poses as input.
Paper Structure (16 sections, 5 equations, 5 figures, 4 tables)

This paper contains 16 sections, 5 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. 3D Human pose estimation
  4. Skeleton-based human action recognition
  5. Multi-task learning
  6. Multi-modal representation learning
  7. Methodology
  8. Network architecture
  9. Pretraining tasks
  10. Fine-tuning
  11. Experiments
  12. Pretraining
  13. Fine-tuning for 3D human pose estimation
  14. Qualitative results on global representation learning
  15. Discussion
  16. ...and 1 more sections

Figures (5)

  • Figure 1: The overall architecture of LangPose.
  • Figure 2: The human body is partitioned into six parts for motion modeling with masked body parts.
  • Figure 3: Masking different segments of a human motion has distinct effects. Top: masking the initial part of the motion. Middle: masking the middle segment. Bottom: masking the latter part of the motion.
  • Figure 4: The t-SNE visualization of global representations of motion embeddings on the BABEL dataset: on the left (a) are embeddings obtained through random joint-level and frame-level masking, as proposed by MotionBERT motionbert2022, with average pooling of all pose features in the sequence; on the right (b) are embeddings obtained through LangPose.
  • Figure 5: Qualitative results of LangPose compared to the baseline model MotionBERT. The left column shows the results from LangPose, the middle column shows the results from MotionBERT for the same frames, and the right column displays the corresponding original 2D pose inputs.