In My Perspective, In My Hands: Accurate Egocentric 2D Hand Pose and Action Recognition

TL;DR

This work tackles egocentric action recognition using only 2D hand pose information extracted from a single RGB image. It introduces two architectures, EffHandNet (top-down single-hand) and EffHandEgoNet (bottom-up egocentric with hand–object interactions), and couples them with a transformer-based action recognizer to achieve state-of-the-art results on H2O ($91.32\%$) and FPHA ($94.43\%$) while delivering fast inference on standard hardware. The approach relies on YOLOv7 for object detection and outputs 2D hand keypoints ($Ph_{2D}^{L}, Ph_{2D}^{R}$) and 2D object pose ($Po_{2D}, Po_l$), enabling robust action understanding without depth sensors. Extensive ablations show that 2D pose accuracy directly influences recognition performance and that the egocentric bottom-up model handles occlusions and hand–object interactions more effectively than non-egocentric 2D methods. The findings highlight the practical potential of 2D skeletal data for real-world egocentric vision, particularly for consumer AR glasses and scenarios where depth sensing is impractical.

Abstract

Action recognition is essential for egocentric video understanding, allowing automatic and continuous monitoring of Activities of Daily Living (ADLs) without user effort. Existing literature focuses on 3D hand pose input, which requires computationally intensive depth estimation networks or wearing an uncomfortable depth sensor. In contrast, there has been insufficient research in understanding 2D hand pose for egocentric action recognition, despite the availability of user-friendly smart glasses in the market capable of capturing a single RGB image. Our study aims to fill this research gap by exploring the field of 2D hand pose estimation for egocentric action recognition, making two contributions. Firstly, we introduce two novel approaches for 2D hand pose estimation, namely EffHandNet for single-hand estimation and EffHandEgoNet, tailored for an egocentric perspective, capturing interactions between hands and objects. Both methods outperform state-of-the-art models on H2O and FPHA public benchmarks. Secondly, we present a robust action recognition architecture from 2D hand and object poses. This method incorporates EffHandEgoNet, and a transformer-based action recognition method. Evaluated on H2O and FPHA datasets, our architecture has a faster inference time and achieves an accuracy of 91.32% and 94.43%, respectively, surpassing state of the art, including 3D-based methods. Our work demonstrates that using 2D skeletal data is a robust approach for egocentric action understanding. Extensive evaluation and ablation studies show the impact of the hand pose estimation approach, and how each input affects the overall performance.

Figures (6)

• Figure 2: Currently, comfortable wearable RGB-D cameras are not readily available in the market. Left: A self-made RGB-D setup Kwon_2021_ICCV. Right: User wearing RayBan glasses with an integrated RGB camerarayban.
• Figure 3: Our EffHandEgoNet architecture to resolve keypoint prediction problem. Input images are resized to 512 pixels and passed through the network to produce 21 heatmaps for each of the hand's keypoints by both upsamplers. The size of the layers is illustrative.
• Figure 4: Our procedure for action recognition. From the sequence of frames $f_1, f_2, f_3 .. f_n$ hand pose $Ph_{2D}^L$,$Ph_{2D}^R$ is estimated using EffHandEgoNet model and object pose $Po_{2D}$, $Po_{l}$ is extracted with YOLOv7wang2022yolov7. Each sequence frame $f_n$ is linearised from shape $\mathbb{R}^{93}$ to $\mathbb{R}^{42}$. With added positional embedding and classification token, this information creates an input for the transformer encoder implemented following dosovitskiy2020image repeated $\times2$ times, which embeds the temporal information. Finally, the multi-layer perceptron predicts one of the 36 action labels.
• Figure 5: Inference time and accuracy per single action of state-of-the-art methods on H2O Dataset. Our method predicts the fastest with the highest accuracy.
• Figure 6: EPE results for different methods in edge scenarios for overlapping and fully separated hands in H2O Dataset.