EgoHandICL: Egocentric 3D Hand Reconstruction with In-Context Learning

TL;DR

EgoHandICL presents the first in-context learning framework for egocentric 3D hand reconstruction by leveraging VLM-guided exemplar retrieval, a multimodal ICL tokenizer, and a MAE-based reconstruction backbone to achieve robust, generalizable hand meshes from monocular RGB data. The method aligns input–output MANO representations across retrieved templates, encodes multimodal context, and learns with masked reconstruction to enable exemplar-conditioned reasoning. Across ARCTIC and EgoExo4D, EgoHandICL delivers consistent improvements in both vertex- and joint-level metrics, handles severe occlusions, and improves spatial consistency in bimanual scenarios. Furthermore, reconstructed hands can enhance EgoVLM-based hand–object interaction reasoning, demonstrating practical value for downstream egocentric tasks and AR/VR applications. Limitations include VLM retrieval overhead and dependence on diverse template banks, with future work exploring faster retrieval, richer annotations, end-to-end benchmarks, and temporal hand tracking extensions.

Abstract

Robust 3D hand reconstruction in egocentric vision is challenging due to depth ambiguity, self-occlusion, and complex hand-object interactions. Prior methods mitigate these issues by scaling training data or adding auxiliary cues, but they often struggle in unseen contexts. We present EgoHandICL, the first in-context learning (ICL) framework for 3D hand reconstruction that improves semantic alignment, visual consistency, and robustness under challenging egocentric conditions. EgoHandICL introduces complementary exemplar retrieval guided by vision-language models (VLMs), an ICL-tailored tokenizer for multimodal context, and a masked autoencoder (MAE)-based architecture trained with hand-guided geometric and perceptual objectives. Experiments on ARCTIC and EgoExo4D show consistent gains over state-of-the-art methods. We also demonstrate real-world generalization and improve EgoVLM hand-object interaction reasoning by using reconstructed hands as visual prompts. Code and data: https://github.com/Nicous20/EgoHandICL

Figures (13)

• Figure 1: Comparing EgoHandICL with previous methods. (i) Prior works improve egocentric hand reconstruction by exploiting auxiliary cues prakash20243dHara2025EventEgoHands, thereby having limited capabilities in handling challenging scenarios with severe occlusions. (ii) EgoHandICL improves the precision through two key steps. Step A: We prompt vision-language models with egocentric cues to retrieve template images for the input query image. Step B: We construct ICL demonstrations by aligning the input-target pairs of both template and query images.
• Figure 2: Overview of our EgoHandICL framework.Part A: Given a query image, we retrieve templates via two complementary strategies. Pre-defined Visual Templates: a VLM classifies the hand-involvement type and retrieves a template image of the same type. Adaptive Textual Templates: we prompt the VLM to generate semantic descriptions, and retrieve a template image given textual similarity. Part B: We encode image tokens $\text{F}_{\text{i}}$, structural tokens $\text{F}_{\text{m}}$, and text tokens $\text{F}_{\text{t}}$, respectively; and then apply cross-attention to tokenize four structured sets of ICL tokens. Part C: We follow a MAE-style design, where the template and query target tokens are partially masked to train the Transformer. In inference, the query target tokens are fully masked for the Transformer's prediction.
• Figure 3: Qualitative results on the ARCTIC dataset. Note: In the bottom case, where the two hands cross and the left hand is severely occluded, WiLoR potamias2025wilor reconstructs only the right hand but mistakenly identifies it as the left.
• Figure 4: Qualitative results on the EgoExo4D dataset (left) and self-captured cases (right). Note: In the bottom-left case with a single heavily occluded hand, HaMeR pavlakos2024reconstructing mistakenly reconstructs two hands, whereas WiLoR potamias2025wilor fails to reconstruct any.
• Figure 4: Comparison of different prompts for adaptive textual templates retrieval. Results are tested on the ARCTIC dataset.