Direction-Aware Hybrid Representation Learning for 3D Hand Pose and Shape Estimation

TL;DR

This work proposes learning direction-aware hybrid features (DaHyF) that fuse implicit image features and explicit 2D joint coordinate features that are enhanced by the pixel direction information in the camera coordinate system to estimate pose, shape, and camera viewpoint.

Abstract

Most model-based 3D hand pose and shape estimation methods directly regress the parametric model parameters from an image to obtain 3D joints under weak supervision. However, these methods involve solving a complex optimization problem with many local minima, making training difficult. To address this challenge, we propose learning direction-aware hybrid features (DaHyF) that fuse implicit image features and explicit 2D joint coordinate features. This fusion is enhanced by the pixel direction information in the camera coordinate system to estimate pose, shape, and camera viewpoint. Our method directly predicts 3D hand poses with DaHyF representation and reduces jittering during motion capture using prediction confidence based on contrastive learning. We evaluate our method on the FreiHAND dataset and show that it outperforms existing state-of-the-art methods by more than 33% in accuracy. DaHyF also achieves the top ranking on both the HO3Dv2 and HO3Dv3 leaderboards for the metric of Mean Joint Error (after scale and translation alignment). Compared to the second-best results, the largest improvement observed is 10%. We also demonstrate its effectiveness in real-time motion capture scenarios with hand position variability, occlusion, and motion blur.

Figures (11)

• Figure 1: Overview of DaHyF. A hand region is cropped, resized and encoded as a local implicit feature map, which is then fused with a global direction map calculated from the hand bounding box. The fused features are used to detect 2D hand keypoints $J_{2d}$ in sub-pixel accuracy, which are positionally encoded (PE) with sinusoidal signals to build a joint feature vector. The pooled implict $F_m$ and the encoded explict $J_{2d}$ are concatenated to form our direction-aware hybrid feature (DaHyF) vector. This DaHyF vector is used to regress hand pose, shape, and camera parameters. Finally, 3D keypoints are obtained based on the MANO model romero2022embodied and projected to 2D coordinates $J_{2d}^{proj}$ for confidence computation with $J_{2d}$.
• Figure 1: 3D PCK on FreiHAND.
• Figure 2: The input image features of $a$ and $b$ are very similar after cropping and resizing the hand patches. From their local direction maps, we can see that they are identical under the input image feature coordinate space. Thus, the local direction maps do not help CNN distinguish $a$ from $b$, and it is difficult to regress the global 3D pose information of the hands with respect to the camera coordinate system.
• Figure 2: 3D PCK on HO3Dv2.
• Figure 3: Global direction map construction. First, we calculate $P_l^i$ from $P_f^i$, then $P_g^i$ from $P_l^i$ and $P_d^i$ (not shown here) from $P_g^i$, and finally form the global feature map from $P_d^i=(x_d^i, y_d^i)$.