DextER: Language-driven Dexterous Grasp Generation with Embodied Reasoning

TL;DR

DextER tackles language-conditioned dexterous grasp generation by introducing contact-based embodied reasoning as an intermediate representation. The method factorizes grasp synthesis with $p(\mathbf{a}, \mathcal{C} | \mathbf{P}, \mathbf{T}) = p(\mathcal{C} | \mathbf{P}, \mathbf{T}) \cdot p(\mathbf{a} | \mathcal{C}, \mathbf{P}, \mathbf{T})$ and autoregressively generates discretized contact tokens before grasp tokens, enabling interpretable reasoning that respects object geometry and hand constraints. Trained on DexGYS and Dexonomy with physics-based contact annotations from MuJoCo and language descriptions from VLMs, DextER achieves state-of-the-art 67.14% grasp success on DexGYS and a 96.4% improvement in intention alignment, while supporting steerable grasp generation via partial contact specification. The approach demonstrates strong generalization to unseen objects and tasks, provides transparent reasoning through explicit contact predictions, and offers a practical interface for fine-grained control over grasp synthesis, albeit with limitations in compounding errors and static-scene evaluation.

Abstract

Language-driven dexterous grasp generation requires the models to understand task semantics, 3D geometry, and complex hand-object interactions. While vision-language models have been applied to this problem, existing approaches directly map observations to grasp parameters without intermediate reasoning about physical interactions. We present DextER, Dexterous Grasp Generation with Embodied Reasoning, which introduces contact-based embodied reasoning for multi-finger manipulation. Our key insight is that predicting which hand links contact where on the object surface provides an embodiment-aware intermediate representation bridging task semantics with physical constraints. DextER autoregressively generates embodied contact tokens specifying which finger links contact where on the object surface, followed by grasp tokens encoding the hand configuration. On DexGYS, DextER achieves 67.14% success rate, outperforming state-of-the-art by 3.83%p with 96.4% improvement in intention alignment. We also demonstrate steerable generation through partial contact specification, providing fine-grained control over grasp synthesis.

Figures (10)

• Figure 1: DextER introduces contact-based embodied reasoning for language-driven dexterous grasp generation. Given a 3D object and instruction, DextER autoregressively predicts which finger links contact where on the object surface before generating the final grasp. Our method achieves state-of-the-art performance with significant improvement in intention alignment and enables steerable generation where users can guide grasp synthesis by specifying partial contact constraints.
• Figure 2: DextER model architecture. Our model processes 3D point clouds and language instructions to predict dexterous grasping actions for the multi-fingered robotic hand. (Left) The input point clouds and textual grasp descriptions are encoded into tokens using a pretrained point cloud encoder liu2025partfield and a text tokenizer qwen2qwen2.5. (Middle) The LLM backbone qwen2qwen2.5 fuses point cloud embeddings with text prompts and autoregressively generates discretized contact and action tokens. (Right) The generated contact and action tokens are de-tokenized into contact positions, hand joint configurations, and grasp poses.
• Figure 3: Qualitative results on language-conditioned dexterous grasp generation. Given object point clouds and natural language instructions, DextER generates embodied contact predictions (shown as colored spheres on object surfaces) followed by grasp configurations. The model successfully captures task-specific contact patterns and produces physically plausible grasps that align with language instructions across diverse objects and manipulation intents. The 3rd and 4th rows show predictions from the same model (DextER), visualized in two separate columns to better highlight the predicted contact points in the 3rd column.
• Figure 4: Prefix-LM attention mask for DextER. Point cloud (PC) tokens use bidirectional attention (full blue blocks in PC rows/columns), whereas the other tokens use causal attention (lower triangular patterns), attending to all preceding point cloud tokens.
• Figure 5: Contact annotation example.