PandaPose: 3D Human Pose Lifting from a Single Image via Propagating 2D Pose Prior to 3D Anchor Space

TL;DR

PandaPose tackles the challenge of lifting 3D human pose from a single image by propagating 2D pose priors into a learnable 3D anchor space. It introduces adaptive joint-wise 3D anchors, depth-aware feature lifting, and an anchor-feature interaction decoder to form unified anchor queries and enable anchor-to-joint ensemble predictions. The approach achieves state-of-the-art performance on Human3.6M, MPI-INF-3DHP, and 3DPW, with notable robustness under occlusion and input noise. By decoupling 2D pose accuracy from final 3D prediction through anchors and depth distribution modeling, PandaPose offers improved robustness and practical applicability for single-image 3D pose estimation in diverse environments.

Abstract

3D human pose lifting from a single RGB image is a challenging task in 3D vision. Existing methods typically establish a direct joint-to-joint mapping from 2D to 3D poses based on 2D features. This formulation suffers from two fundamental limitations: inevitable error propagation from input predicted 2D pose to 3D predictions and inherent difficulties in handling self-occlusion cases. In this paper, we propose PandaPose, a 3D human pose lifting approach via propagating 2D pose prior to 3D anchor space as the unified intermediate representation. Specifically, our 3D anchor space comprises: (1) Joint-wise 3D anchors in the canonical coordinate system, providing accurate and robust priors to mitigate 2D pose estimation inaccuracies. (2) Depth-aware joint-wise feature lifting that hierarchically integrates depth information to resolve self-occlusion ambiguities. (3) The anchor-feature interaction decoder that incorporates 3D anchors with lifted features to generate unified anchor queries encapsulating joint-wise 3D anchor set, visual cues and geometric depth information. The anchor queries are further employed to facilitate anchor-to-joint ensemble prediction. Experiments on three well-established benchmarks (i.e., Human3.6M, MPI-INF-3DHP and 3DPW) demonstrate the superiority of our proposition. The substantial reduction in error by $14.7\%$ compared to SOTA methods on the challenging conditions of Human3.6M and qualitative comparisons further showcase the effectiveness and robustness of our approach.

Figures (11)

• Figure 1: Comparison between different 2D-to-3D human pose lifting manners. Previous methods (a) generally concern 2D in-plane feature and directly predict 3D pose. Our method (b) mitigates depth ambiguity by lifting in-plane feature to 3D space and interacting with joint-wise 3D anchors. Then 3D pose will be estimated via anchor-to-joint ensemble prediction within 3D anchor space. Our method is robust to both indoor and outdoor occlusion scenarios.
• Figure 2: Overview of PandaPose pipeline. Given input single-frame 2D pose and intermediate image features, we adaptively sample anchors in 3D space. By estimating joint-wise depth distributions and employing a 2D pose prior based sampling strategy, we lift features from 2D to 3D domain. After 3D anchor-feature interaction, we obtain predicted 3D pose through anchor-to-joint ensemble prediction.
• Figure 3: Comparison between different anchor settings. Adaptive anchor is more effective at joints distant from human regions.
• Figure 4: Illustration of adaptive 3D anchor setting including joint-wise local anchor and global fixed anchor.
• Figure 5: Existing methods typically predict a single depth map. Due to projection distortion, joints far apart in 3D space may appear close in 2D planes, often overlapping in the downsampled feature map, causing significant confusion during training and inference.