Towards Egocentric 3D Hand Pose Estimation in Unseen Domains

TL;DR

V-HPOT tackles cross-domain egocentric 3D hand pose estimation by learning depth in a virtual camera space that decouples depth from camera intrinsics and by employing a self-supervised test-time optimisation that enforces 3D spatial consistency under depth augmentations. The method introduces pseudodepth as an auxiliary training task and a 3D consistency loss during inference, updating the feature extractor online for unseen domains without ground-truth labels. Empirical results show large reductions in absolute pose error ($MPJPE$ and $MRRPE$) across H2O and AssemblyHands, with competitive performance on in-the-wild Epic-Kpts and favorable comparisons to both single-stage and two-stage baselines while using far less data. The approach demonstrates strong cross-domain generalisation, robustness to intrinsics variation, and practical applicability to real-world egocentric hand pose tasks, including AR/VR and robotics contexts. Overall, V-HPOT offers a data-efficient, inference-time adaptable solution for reliable 3D hand pose estimation in unseen environments.

Abstract

We present V-HPOT, a novel approach for improving the cross-domain performance of 3D hand pose estimation from egocentric images across diverse, unseen domains. State-of-the-art methods demonstrate strong performance when trained and tested within the same domain. However, they struggle to generalise to new environments due to limited training data and depth perception -- overfitting to specific camera intrinsics. Our method addresses this by estimating keypoint z-coordinates in a virtual camera space, normalised by focal length and image size, enabling camera-agnostic depth prediction. We further leverage this invariance to camera intrinsics to propose a self-supervised test-time optimisation strategy that refines the model's depth perception during inference. This is achieved by applying a 3D consistency loss between predicted and in-space scale-transformed hand poses, allowing the model to adapt to target domain characteristics without requiring ground truth annotations. V-HPOT significantly improves 3D hand pose estimation performance in cross-domain scenarios, achieving a 71% reduction in mean pose error on the H2O dataset and a 41% reduction on the AssemblyHands dataset. Compared to state-of-the-art methods, V-HPOT outperforms all single-stage approaches across all datasets and competes closely with two-stage methods, despite needing approximately x3.5 to x14 less data.

Figures (6)

• Figure 1: V-HPOT improves egocentric 3D hand pose estimation in cross-domain scenarios, where a model is trained in source and tested in different target domain. We propose training the model to estimate the $z$-coordinate in virtual camera space to mitigate overfitting to source domain camera intrinsics. At test time, we optimise predictions using a novel self-supervised loss that enforces 3D spatial consistency, allowing the model to refine depth perception by aligning depth-augmented poses in 3D camera space. This significantly improves absolute 3D pose error (MPJPE) without any labelled data, outperforming existing methods focused solely on 2D improvement.
• Figure 2: V-HPOT: We train in source domain to estimate 3D hand pose $P_{L,R}^{3D}$ from an image $I_n^{RGB}$ in virtual camera space, with an auxiliary pseudo-depth task $I_n^D$ supervised by DPT-Hybridranftl2021vision. At test time, the model predicts an initial pose $P_{L,R}^{\text{init}}$ and applies depth augmentation by scaling $I_n^{RGB}$ with $S_i$ in virtual camera space, yielding $P_{L,R}^{\rightarrow S}$. A 3D consistency loss between these poses refines the backbone, enhancing final predictions.
• Figure 3: Visualisation of L2 loss for $x, y$ and L1 loss for the $z$ coordinate over test samples in cross-domain scenarios. The dashed line represents 5% of data where V-HPOT stops optimising. Continuous improvement is observed for most samples.
• Figure 4: Qualitative results of our method in 2D and 3D space. Green skeletons represent the ground truth hand pose, red estimations withoutV-HPOT and blue estimations withV-HPOT. Four examples from the left (a-d) show that V-HPOT improves 3D pose estimation, while two examples from the right side (e) present a negative case with an increase of error.
• Figure 5: Qualitative results of our method on the in-the-wild Epic-Kpts and own images. The estimated 3D pose is projected into 2D space, as no ground truth 3D pose labels are available. Despite the challenging natural environments full of everyday objects, various backgrounds, and sceneries, V-HPOT produces geometrically accurate poses. In Epic-Kpts, camera wearers perform dynamic actions involving manipulating various objects while preparing meals or cleaning. In addition to Epic-Kpts, we capture images without depth cues, featuring an outdoor background and a subject pointing at the wall, without any reference points.