DeltaDorsal: Enhancing Hand Pose Estimation with Dorsal Features in Egocentric Views

TL;DR

DeltaDorsal tackles the challenge of egocentric hand pose estimation under frequent self-occlusion by leveraging dorsal skin deformation. The authors propose a dual-stream delta encoder built on a DINOv3 backbone to compare dorsal features from a current pose against a neutral reference, enabling accurate pose prediction with purely dorsal cues. They introduce a high-resolution 4K dorsal dataset collected from 12 participants across 17 gestures and demonstrate that DeltaDorsal outperforms state-of-the-art baselines in occluded scenarios, while remaining more compact and efficient. The approach also enables downstream interactions such as tap, pinch, and isometric force click, suggesting practical XR applications and potential for mobile deployment. Overall, the work provides a new sensing modality that complements silhouette-based methods and broadens the robustness and expressiveness of monocular, ego-centric hand tracking.

Abstract

The proliferation of XR devices has made egocentric hand pose estimation a vital task, yet this perspective is inherently challenged by frequent finger occlusions. To address this, we propose a novel approach that leverages the rich information in dorsal hand skin deformation, unlocked by recent advances in dense visual featurizers. We introduce a dual-stream delta encoder that learns pose by contrasting features from a dynamic hand with a baseline relaxed position. Our evaluation demonstrates that, using only cropped dorsal images, our method reduces the Mean Per Joint Angle Error (MPJAE) by 18% in self-occluded scenarios (fingers >= 50% occluded) compared to state-of-the-art techniques that depend on the whole hand's geometry and large model backbones. Consequently, our method not only enhances the reliability of downstream tasks like index finger pinch and tap estimation in occluded scenarios but also unlocks new interaction paradigms, such as detecting isometric force for a surface "click" without visible movement while minimizing model size.

Figures (10)

• Figure 1: a) Manually defined OpenPose keypoint alignment on the MANO model caoOpenPoseRealtimeMultiPerson2019a. Minor deviations from the mesh are caused by MANO's internal definition of a joint position. b) Categorization of MANO mesh faces. Each color represents a separate categorization corresponding to one of index, middle, ring, pinky, thumb, dorsum, and palm.
• Figure 2: a) Number of fingers with over 90% of surface faces visible across different datasets. b) Distribution of dorsal visibility when atleast one finger is over 90% occluded across different datasets. The internal box represents the 25th to 75th percentile while the white line represents the median.
• Figure 3: MPJAE compared to average per finger visibility in our unseen dataset. The red line denotes the fitted linear regression for visualized data. Mean finger visibility is defined as the percentage of face surface area that is visible to the camera averaged across all five fingers. The higher the MPJAE, the worse the performance of the model is.
• Figure 4: Examples of each static gesture collected in our data collection. Not depicted are the two dynamic gestures: fanning (30s) and freeform (60s). On the top are the following: An aligned image of the reference, the picture of the dorsal features during this gesture, and the cosine similarity mapping for the DINO features generated from the reference and the current image. The color of the similarity map indicates a smaller cosine similarity (darker is more different). (I: index finger, M: middle finger, R: ring finger, P: pinky, T: thumb).
• Figure 5: DeltaDorsal's system architecture. Users input a "reference" image of their hand in a neutral position and a picture of the hand in some gesture. An initial hand pose prediction from HaMeR is then used to align both hands so that their dorsal features are spatially localized. Images of dorsal features are then fed into DINOv3 to extract image features. These features, along with the cosine similarity and difference between the "reference" and current image's features, are fed into the change encoder. A regression head then predicts the current hand pose, which can be processed with MANO using a prior shape prediction to generate a hand mesh. Optionally, users can use the initial translation prediction from HaMeR to localize the final mesh in the camera frame.