Language-Guided Grasp Detection with Coarse-to-Fine Learning for Robotic Manipulation

TL;DR

LGGD addresses the challenge of grounding robotic grasping in natural language by embedding linguistic intent throughout the perception pipeline. It introduces a coarse-to-fine architecture with a dual cross vision-language fusion bottleneck (DCVLF), a language-guided upsampling module (LMAFN), and a language-conditioned dynamic convolution head (LDCH), enabling fine-grained, instruction-consistent grasp predictions. Evaluations on OCID-VLG and Grasp-Anything++ show state-of-the-art performance and strong generalization to unseen objects and novel prompts, with real-robot experiments confirming practical viability. The work demonstrates that deep, multi-stage vision-language reasoning substantially improves semantic grounding and manipulation reliability in cluttered, real-world environments.

Abstract

Grasping is one of the most fundamental challenging capabilities in robotic manipulation, especially in unstructured, cluttered, and semantically diverse environments. Recent researches have increasingly explored language-guided manipulation, where robots not only perceive the scene but also interpret task-relevant natural language instructions. However, existing language-conditioned grasping methods typically rely on shallow fusion strategies, leading to limited semantic grounding and weak alignment between linguistic intent and visual grasp reasoning.In this work, we propose Language-Guided Grasp Detection (LGGD) with a coarse-to-fine learning paradigm for robotic manipulation. LGGD leverages CLIP-based visual and textual embeddings within a hierarchical cross-modal fusion pipeline, progressively injecting linguistic cues into the visual feature reconstruction process. This design enables fine-grained visual-semantic alignment and improves the feasibility of the predicted grasps with respect to task instructions. In addition, we introduce a language-conditioned dynamic convolution head (LDCH) that mixes multiple convolution experts based on sentence-level features, enabling instruction-adaptive coarse mask and grasp predictions. A final refinement module further enhances grasp consistency and robustness in complex scenes.Experiments on the OCID-VLG and Grasp-Anything++ datasets show that LGGD surpasses existing language-guided grasping methods, exhibiting strong generalization to unseen objects and diverse language queries. Moreover, deployment on a real robotic platform demonstrates the practical effectiveness of our approach in executing accurate, instruction-conditioned grasp actions. The code will be released publicly upon acceptance.

Figures (15)

• Figure 1: Overview of our system: An RGB-D camera mounted on the robot's wrist captures visual data of objects to be grasped. Our proposed LGGD generates 4-DoF grasp poses based on the RGB image and language query during the inference process. These generated grasp poses are then utilized by the control module to plan and execute robot trajectories for pick-and-place tasks.
• Figure 2: Rectangular grasp representation.
• Figure 3: An overview of our proposed LGGD framework. Given an RGB image and a natural-language command, a CLIP-based image encoder and text encoder extract visual features and word/sentence embeddings. The Dual Cross Vision-Language Fusion (DCVLF) bottleneck aligns the two modalities, after which hierarchical language-guided upsampling progressively refines spatial details according to the textual intent. A coarse mask and grasp prediction head outputs the segmentation mask, grasp quality, angle, and gripper width. Finally, the mask refinement and grasp refinement modules sharpen boundaries and stabilize grasp poses, producing accurate, instruction-consistent grasp poses.
• Figure 4: Structure of Image Encoder. The CLIP-based ResNet-50 backbone extracts progressively abstracted feature maps across four residual stages, while attention pooling enhances global semantic perception.
• Figure 5: Structure of the DCVLF. Visual features are enriched with positional encoding and refined by a self-attention block. Bidirectional cross-attention then allows image features to attend to language cues and textual features to attend to visual regions, achieving full semantic alignment. Residual connections and FFNs stabilize learning, while a subsequent $1{\times}1$ Conv and global MHSA-FFN block further integrate local and global context, yielding robust multimodal representations for grasp reasoning.