On the Utility of 3D Hand Poses for Action Recognition

TL;DR

This paper addresses the challenge of action recognition from 3D hand poses by proposing HandFormer, a lightweight multimodal transformer that processes dense 3D hand pose sequences at high temporal resolution while sampling a small number of RGB frames to capture scene semantics. The architecture decomposes actions into micro-actions, encodes per-joint hand trajectories with a Trajectory Encoder, fuses pose and RGB through a Multimodal Tokenizer, and aggregates information via a Temporal Transformer, trained with a combination of classification, verb/object supervision, and feature-anticipation losses. Key findings show that HandFormer achieves state-of-the-art results on Assembly101 and H2O, with pose-only variants outperforming existing skeleton-based methods at around 5× fewer FLOPs, and that incorporating sparse RGB frames significantly improves object recognition and egocentric action understanding. The work demonstrates strong cross-view generalization, efficiency advantages over dense RGB video models, and robust ablations that highlight the value of micro-action factorization, joint trajectory encoding, and multimodal fusion for hand–object interaction recognition.

Abstract

3D hand pose is an underexplored modality for action recognition. Poses are compact yet informative and can greatly benefit applications with limited compute budgets. However, poses alone offer an incomplete understanding of actions, as they cannot fully capture objects and environments with which humans interact. We propose HandFormer, a novel multimodal transformer, to efficiently model hand-object interactions. HandFormer combines 3D hand poses at a high temporal resolution for fine-grained motion modeling with sparsely sampled RGB frames for encoding scene semantics. Observing the unique characteristics of hand poses, we temporally factorize hand modeling and represent each joint by its short-term trajectories. This factorized pose representation combined with sparse RGB samples is remarkably efficient and highly accurate. Unimodal HandFormer with only hand poses outperforms existing skeleton-based methods at 5x fewer FLOPs. With RGB, we achieve new state-of-the-art performance on Assembly101 and H2O with significant improvements in egocentric action recognition.

Figures (10)

• Figure 1: We densely sample 3D hand poses to understand fine-grained hand motions and sparsely sample RGB frames to capture the scene semantics. Our 3D hand poses are acquired from low-resolution monochrome cameras but can also come from wearable sensors, facilitating an efficient understanding of hand-object interactions. Video frames and hand poses in the figure are from Assembly101 sener2022assembly101.
• Figure 2: Comparing skeletal changes in full-body actions from NTU RGB+D 120 liu2019ntu (left) and hand actions from Assembly101 sener2022assembly101 (right). Two pose frames at interval $T$ are shown. $J_{j,t}$ indicates the 3D coordinate of joint $j$ at timestep $t$, and ${<}J_x,J_y{>}$ is the correlation between two such joints. Modeling the correlation between spatio-temporally distant joints can be informative for full-body poses but does not provide a useful action cue for hand poses.
• Figure 3: Overall architecture of HandFormer. An action segment is divided into $K$ micro-actions $\{M_1, M_2 \ldots M_K\}$. Each micro-action comprises a dense sequence of pose frames and a single RGB frame. The frame encoder $F$ and the trajectory encoder $T$ encode the RGB and the dense poses, respectively, after which they are passed to a Multimodal Tokenizer. The modality-mixed tokens are then fed to a Temporal Transformer. Dotted paths are optional and only required when RGB is used.
• Figure 4: Our Trajectory Encoder $T$, which operates on micro-actions, derives tokens with trajectory-based features and performs self-attention to encode the pose sequence into a feature vector. The Single-Joint TCN is a Temporal Convolutional Network lea2017temporal that processes the trajectories of all the joints individually with shared parameters across all joints. Wrist-TCN takes an action-wide sequence of wrist location and hand orientation (6D pose) to produce a global reference token.
• Figure 4: Egocentric action recognition in Assembly101 sener2022assembly101. TSM features from sener2022assembly101 are used as RGB frame features.