3D Human Pose Estimation via Spatial Graph Order Attention and Temporal Body Aware Transformer

TL;DR

This work tackles 3D human pose estimation from 2D sequences by merging spatial graph reasoning with temporal attention. It introduces MO-GCN to capture multi-order spatial relationships, GOA to weight the relevance of each order per joint, and BAT to model joint-specific temporal evolution while emphasizing central frames via BCMA. The combination yields state-of-the-art or competitive results on Human3.6m, MPI-INF-3DHP, and HumanEva-I, with ablations confirming the contributions of each module. The approach offers a scalable framework that leverages graph-based spatial modeling and targeted temporal attention to improve pose accuracy in diverse settings.

Abstract

Nowadays, Transformers and Graph Convolutional Networks (GCNs) are the prevailing techniques for 3D human pose estimation. However, Transformer-based methods either ignore the spatial neighborhood relationships between the joints when used for skeleton representations or disregard the local temporal patterns of the local joint movements in skeleton sequence modeling, while GCN-based methods often neglect the need for pose-specific representations. To address these problems, we propose a new method that exploits the graph modeling capability of GCN to represent each skeleton with multiple graphs of different orders, incorporated with a newly introduced Graph Order Attention module that dynamically emphasizes the most representative orders for each joint. The resulting spatial features of the sequence are further processed using a proposed temporal Body Aware Transformer that models the global body feature dependencies in the sequence with awareness of the local inter-skeleton feature dependencies of joints. Given that our 3D pose output aligns with the central 2D pose in the sequence, we improve the self-attention mechanism to be aware of the central pose while diminishing its focus gradually towards the first and the last poses. Extensive experiments on Human3.6m, MPIINF-3DHP, and HumanEva-I datasets demonstrate the effectiveness of the proposed method. Code and models are made available on Github.

Figures (9)

• Figure 1: Framework of the proposed method. Multiple-order GCN (MO-GCN) generates features of various graph orders for each skeleton. The graph Order Attention (GOA) module assigns weights to various orders for each joint within each pose. The temporal Body Aware Transformer (BAT) captures local attention among body joints using Joints Weighted Attention (JWA) and incorporates global-centered attention using Body-Centred Multi-head Attention (BCMA).
• Figure 2: Visual illustration of the proposed Graph Order Attention module. For each skeleton in the input sequence, each joint is represented with features of the highest attention weight of the same frame along different orders. Big joints represent high attention weights.
• Figure 3: Illustration of the distance calculation between the central frame and the left/right poses in the sequence. $Pos = [P_1, \dots, P_c, \dots, P_T]$, $Dis = [Dis_1, \dots, Dis_c, \dots, Dis_T]$, and $F_{pos} * \textbf{W}^{F}$ correspond to those defined in equations \ref{['eq:dis']} and \ref{['eq:Pscl']}.
• Figure 4: Detailed structure of GOA, JWA, and BCMA. T: Number of frames, J: Number of joints, R: Number of the highest order, D: Features dimension. Pscl: Scaling position vector in Equation \ref{['eq:Pscl']}. The colors match the components in Fig. \ref{['fig:frame']}. Q, K (red) correspond to the symbols in Equation \ref{['eq:3']}, and Q, K, V (blue) correspond to the symbols in Equation \ref{['eq:13']}.
• Figure 5: Ablation study on the effect of the number of orders R on the performance using 9 frames of 2D ground truth (GT) joints from HumanEva-I dataset on $Protocol\#1$ for 8xBAT.