TOUCH: Text-guided Controllable Generation of Free-Form Hand-Object Interactions

TL;DR

This work addresses generating diverse, controllable hand-object interactions beyond grasping by introducing Free-Form HOI Generation. It presents TOUCH, a three-stage diffusion framework with explicit contact modeling, multi-level semantic conditioning, and a physical refinement step to ensure realism. To support this, WildO2 provides in-the-wild 3D HOI data with rich annotations and DSCs, built via an automated O2HOI pipeline. Experiments show TOUCH surpasses baselines in contact accuracy, physical plausibility, diversity, and semantic consistency, enabling realistic daily HOI synthesis for AR/VR, robotics, and embodied AI.

Abstract

Hand-object interaction (HOI) is fundamental for humans to express intent. Existing HOI generation research is predominantly confined to fixed grasping patterns, where control is tied to physical priors such as force closure or generic intent instructions, even when expressed through elaborate language. Such an overly general conditioning imposes a strong inductive bias for stable grasps, thus failing to capture the diversity of daily HOI. To address these limitations, we introduce Free-Form HOI Generation, which aims to generate controllable, diverse, and physically plausible HOI conditioned on fine-grained intent, extending HOI from grasping to free-form interactions, like pushing, poking, and rotating. To support this task, we construct WildO2, an in-the-wild diverse 3D HOI dataset, which includes diverse HOI derived from internet videos. Specifically, it contains 4.4k unique interactions across 92 intents and 610 object categories, each with detailed semantic annotations. Building on this dataset, we propose TOUCH, a three-stage framework centered on a multi-level diffusion model that facilitates fine-grained semantic control to generate versatile hand poses beyond grasping priors. This process leverages explicit contact modeling for conditioning and is subsequently refined with contact consistency and physical constraints to ensure realism. Comprehensive experiments demonstrate our method's ability to generate controllable, diverse, and physically plausible hand interactions representative of daily activities. The project page is $\href{https://guangyid.github.io/hoi123touch}{here}$.

Figures (15)

• Figure 1: Overview. We extend HOI generation beyond laboratory “grasp” settings (left) toward broader daily HOI modalities (right), enabling the modeling of more human-like interactions. Our dataset WildO2, built from Internet videos, covers more contacts, more objects, and more actions, and is enriched with descriptive synthetic captions (DSCs) to support fine-grained semantic controllable HOI generation with our method, TOUCH.
• Figure 2: The proposed data generation pipeline for WildO2. The process begins with O2HOI frame pair extraction from in-the-wild videos, followed by a three-stage pipeline for 3D reconstruction, camera alignment, and hand-object refinement that produces high-fidelity interaction data.
• Figure 2: Analyzes the contributions of various components, including the absence of $\mathbf{M}_O$ and $\mathbf{M}_H$ (hoc.) for guiding spatial relationship generation, the multi-level network structure (mul.), and the multi-level text.
• Figure 4: Overview of our three-stage framework TOUCH for generating hand-object interactions from multi-level text prompts and object meshes. CIM stands for the Condition Injection Module.
• Figure 5: Visualization Results. Comparisons of different methods on samples from the WildO2 test set. Each sample consists of SSCs and an object mesh as input, with the output being an interactive hand pose. The last row shows the original authentic 2D HOI frame from internet videos.