NL2Contact: Natural Language Guided 3D Hand-Object Contact Modeling with Diffusion Model

TL;DR

NL2Contact tackles controllable 3D hand–object contact modeling from natural language by introducing a cross-modal latent diffusion framework and the ContactDescribe dataset. A text-to-hand-object fusion module integrates language with hand and object geometry, followed by a two-stage diffusion process that first generates a hand pose and then a contact map conditioned on language and geometry, with a subsequent contact-optimization step. Experiments on ContactDescribe and HO3D show improved grasp pose optimization, realistic contact, and diverse human grasp generation, illustrating strong generalization to unseen objects. This work enables precise, language-driven control of hand–object interactions and opens pathways for interactive manipulation and avatar-controlled synthesis using natural language prompts.

Abstract

Modeling the physical contacts between the hand and object is standard for refining inaccurate hand poses and generating novel human grasp in 3D hand-object reconstruction. However, existing methods rely on geometric constraints that cannot be specified or controlled. This paper introduces a novel task of controllable 3D hand-object contact modeling with natural language descriptions. Challenges include i) the complexity of cross-modal modeling from language to contact, and ii) a lack of descriptive text for contact patterns. To address these issues, we propose NL2Contact, a model that generates controllable contacts by leveraging staged diffusion models. Given a language description of the hand and contact, NL2Contact generates realistic and faithful 3D hand-object contacts. To train the model, we build \textit{ContactDescribe}, the first dataset with hand-centered contact descriptions. It contains multi-level and diverse descriptions generated by large language models based on carefully designed prompts (e.g., grasp action, grasp type, contact location, free finger status). We show applications of our model to grasp pose optimization and novel human grasp generation, both based on a textual contact description.

Figures (6)

• Figure 1: Illustration of the NL2Contact setting. Our method learns to model 3D hand-object contact from language description. The modeling contact can be used to grasp pose optimization. By utilizing the predicted contact to refine the initial grasp, our performance surpasses that of ContactOpt grady2021contactopt and ContactPose brahmbhatt2020contactpose. It's worth noting that ContactPose is the annotation from the motion capture system.
• Figure 2: An example of the dataset with multi-level language descriptions. Given the proposed hand-centered contact prompts, our dataset leverages ChatGPT3 chatgpt to generate multi-level and diverse free-form text descriptions of contact patterns.
• Figure 3: NL2Contact pipeline. We propose a novel method to model 3D hand-object contact using language description. Our framework is composed of 1) a Text-to-Hand-Object Fusion module, efficiently fusing the geometry information from both point clouds and semantic information from natural language, 2) a staged latent diffusion model to first generate the hand pose fitting the hand description and then generate the 3D hand-object contact map conditioned on the descriptions and the generated hand pose. 3) contact optimization, which iteratively optimizes the hand pose using the generated contact. We also added a switch to toggle between its application for grasp pose optimization and human grasp generation.
• Figure 4: We conduct qualitative comparisons with state-of-the-art methods. Our method generates accurate and controllable contact, matching the input text description. ContactOpt grady2021contactopt and S$^2$Contact tse2022s are uncontrollable contact generation methods. They tend to generate a grasp with all five fingers engaged. Interestingly, we provide a great initial pose in the first row. However, ContactOpt and S$^2$Contact still generate a five-figner grasp which is different from the initial pose.
• Figure 5: Visualization of diverse grasp generation. We observe ContactGen liu2023contactgen always generates grasps with all five fingers engaged, while our method generates grasps consistent with the text.