Generative Hierarchical Temporal Transformer for Hand Pose and Action Modeling

TL;DR

The paper addresses the challenge of simultaneously recognizing hand pose and predicting future hand motion by introducing G-HTT, a Generative Hierarchical Temporal Transformer with two cascaded VAEs: a short-span Pose (P) block and a long-span Action (A) block bridged by a mid-level representation $\mathbf{m}$. This semantic-temporal hierarchy allows decoupled training on datasets with different annotations while enforcing global consistency between past and future motions. Empirical results on H2O, Assembly101, and AssemblyHands show that joint recognition and prediction with hierarchical modeling yields superior pose refinement, action recognition, and long-term motion generation compared to baselines like HTT and PoseGPT, and that the mid-level representation improves diversity and fidelity. The approach has practical impact for real-time human-robot interaction and VR/AR applications by enabling coherent, action-conditioned hand motion synthesis across varied data sources and viewpoints.

Abstract

We present a novel unified framework that concurrently tackles recognition and future prediction for human hand pose and action modeling. Previous works generally provide isolated solutions for either recognition or prediction, which not only increases the complexity of integration in practical applications, but more importantly, cannot exploit the synergy of both sides and suffer suboptimal performances in their respective domains. To address this problem, we propose a generative Transformer VAE architecture to model hand pose and action, where the encoder and decoder capture recognition and prediction respectively, and their connection through the VAE bottleneck mandates the learning of consistent hand motion from the past to the future and vice versa. Furthermore, to faithfully model the semantic dependency and different temporal granularity of hand pose and action, we decompose the framework into two cascaded VAE blocks: the first and latter blocks respectively model the short-span poses and long-span action, and are connected by a mid-level feature representing a sub-second series of hand poses. This decomposition into block cascades facilitates capturing both short-term and long-term temporal regularity in pose and action modeling, and enables training two blocks separately to fully utilize datasets with annotations of different temporal granularities. We train and evaluate our framework across multiple datasets; results show that our joint modeling of recognition and prediction improves over isolated solutions, and that our semantic and temporal hierarchy facilitates long-term pose and action modeling.

Figures (6)

• Figure 1: Jointly modeling recognition and prediction, while following the semantic dependency and temporal granularity for hand pose-action. For recognition, (1)$\to$(2) moves up from short to long spans for input pose refinement and action recognition respectively. For motion prediction, two paths are available: (1)$\to$(4) exploits short-term motion regularity, and (1)$\to$(2)$\to$(3)$\to$(4) enables long-term action-guided prediction.
• Figure 2: Overview of our framework. The cascaded $\vb{P}$ and $\vb{A}$ (shaded in blue) of G-HTT jointly model recognition and prediction, and faithfully respect the semantic dependency and temporal granularity among pose, mid-level and action (\ref{['sec:method_hierahrcy']}).
• Figure 3: Qualitative comparison of pose estimation for HTT wen2023hierarchical and ours, on camera view v1 of Assembly datasets ohkawa2023assemblyhandssener2022assembly101. More cases are provided in the supplementary.
• Figure 4: Qualitative comparison of predicted motions for PoseGPT lucas2022posegpt, the ablated settings of w/o mid-level, w/ only $\vb{P}$ via path P.a, and the full G-HTT (w/ $\vb{P},\vb{A}$, via path P.b) on H2O. More qualitative cases are provided in the supplementary.
• Figure 5: All camera views of H2O (upper row), and fixed camera views of the Assembly101/AssemblyHands (lower row). Views denoted in † (i.e.v5,v7) are not leveraged due to frequent severe occlusion.