Hand2World: Autoregressive Egocentric Interaction Generation via Free-Space Hand Gestures

TL;DR

Hand2World, a unified autoregressive framework that addresses egocentric interaction generation from a single scene image under free-space hand gestures through occlusion-invariant hand conditioning based on projected 3D hand meshes, allowing visibility and occlusion to be inferred from scene context rather than encoded in the control signal.

Abstract

Egocentric interactive world models are essential for augmented reality and embodied AI, where visual generation must respond to user input with low latency, geometric consistency, and long-term stability. We study egocentric interaction generation from a single scene image under free-space hand gestures, aiming to synthesize photorealistic videos in which hands enter the scene, interact with objects, and induce plausible world dynamics under head motion. This setting introduces fundamental challenges, including distribution shift between free-space gestures and contact-heavy training data, ambiguity between hand motion and camera motion in monocular views, and the need for arbitrary-length video generation. We present Hand2World, a unified autoregressive framework that addresses these challenges through occlusion-invariant hand conditioning based on projected 3D hand meshes, allowing visibility and occlusion to be inferred from scene context rather than encoded in the control signal. To stabilize egocentric viewpoint changes, we inject explicit camera geometry via per-pixel Plücker-ray embeddings, disentangling camera motion from hand motion and preventing background drift. We further develop a fully automated monocular annotation pipeline and distill a bidirectional diffusion model into a causal generator, enabling arbitrary-length synthesis. Experiments on three egocentric interaction benchmarks show substantial improvements in perceptual quality and 3D consistency while supporting camera control and long-horizon interactive generation.

Figures (16)

• Figure 1: Given a scene image (left) and free-space hand gestures from an egocentric monocular stream (middle), Hand2World synthesizes interaction videos (right) in which hands enter the depicted scene and manipulate objects while following the input viewpoint changes. Each scene is driven by two consecutive gesture rounds and generated autoregressively over 400+ frames, demonstrating arbitrary-length rollout.
• Figure 2: Hand2World encodes gestures as projected 3D hand meshes (silhouette-and-wireframe) to decouple geometry from visibility, and models camera motion via per-pixel Plücker-ray embeddings (left). During training (center), scene and hand controls are channel-concatenated into the diffusion transformer backbone (Wan2.1-1.3B-Control) while camera embeddings are injected additively through a lightweight adapter. The bidirectional teacher is distilled into a causal AR generator via CausVid yin2025causvid and self-forcing huang2025selfforcing (right). At inference (bottom), monocular estimators (HaMeR HaMeR, Depth Anything V3 depthanything3) and KV-cached block-wise generation enable arbitrary-length egocentric interaction from a single scene image and free-space gestures.
• Figure 3: Training videos contain partially visible hands due to object occlusions (top), producing partial masks at training time. At inference, free-space gestures yield fully visible hands and complete masks (bottom), creating a train--test distribution gap. Mask-conditioned methods such as CosHand degrade under this shift, whereas our projected 3D hand mesh maintains a format-consistent control signal regardless of occlusion state.
• Figure 4: Without explicit camera conditioning (top), the model fails to follow the rightward-then-leftward head pan and produces background drift. With Plücker-ray injection (middle), the generated trajectory closely matches the ground truth (bottom).
• Figure 5: Each column compares the full annotation pipeline (top) against a single-component ablation (bottom) at a representative frame. Green boxes highlight affected regions.