Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language-driven Grasp Detection

An Dinh Vuong, Minh Nhat Vu, Baoru Huang, Nghia Nguyen, Hieu Le, Thieu Vo, Anh Nguyen

TL;DR

This work tackles language-conditioned grasp detection by introducing Grasp-Anything++, a large-scale, foundation-model–synthesized dataset with 1M images and 10M grasp prompts, enabling language-driven control of grasping. It also presents LGD, a diffusion-model–based framework that fuses image and language cues via ALBEF and a novel contrastive loss, yielding improved grasp pose generation under natural language instructions. The approach achieves state-of-the-art performance on the dataset, demonstrates zero-shot generalization across unseen objects, and validates practical robotic grasping in real-world experiments. Overall, the dataset and method establish a scalable foundation for integrating natural language with low-level robotic manipulation, with broad potential for future 3D, scene understanding, and human–robot interaction research.

Abstract

Grasp detection is a persistent and intricate challenge with various industrial applications. Recently, many methods and datasets have been proposed to tackle the grasp detection problem. However, most of them do not consider using natural language as a condition to detect the grasp poses. In this paper, we introduce Grasp-Anything++, a new language-driven grasp detection dataset featuring 1M samples, over 3M objects, and upwards of 10M grasping instructions. We utilize foundation models to create a large-scale scene corpus with corresponding images and grasp prompts. We approach the language-driven grasp detection task as a conditional generation problem. Drawing on the success of diffusion models in generative tasks and given that language plays a vital role in this task, we propose a new language-driven grasp detection method based on diffusion models. Our key contribution is the contrastive training objective, which explicitly contributes to the denoising process to detect the grasp pose given the language instructions. We illustrate that our approach is theoretically supportive. The intensive experiments show that our method outperforms state-of-the-art approaches and allows real-world robotic grasping. Finally, we demonstrate our large-scale dataset enables zero-short grasp detection and is a challenging benchmark for future work. Project website: https://airvlab.github.io/grasp-anything/

Language-driven Grasp Detection

TL;DR

This work tackles language-conditioned grasp detection by introducing Grasp-Anything++, a large-scale, foundation-model–synthesized dataset with 1M images and 10M grasp prompts, enabling language-driven control of grasping. It also presents LGD, a diffusion-model–based framework that fuses image and language cues via ALBEF and a novel contrastive loss, yielding improved grasp pose generation under natural language instructions. The approach achieves state-of-the-art performance on the dataset, demonstrates zero-shot generalization across unseen objects, and validates practical robotic grasping in real-world experiments. Overall, the dataset and method establish a scalable foundation for integrating natural language with low-level robotic manipulation, with broad potential for future 3D, scene understanding, and human–robot interaction research.

Abstract

Grasp detection is a persistent and intricate challenge with various industrial applications. Recently, many methods and datasets have been proposed to tackle the grasp detection problem. However, most of them do not consider using natural language as a condition to detect the grasp poses. In this paper, we introduce Grasp-Anything++, a new language-driven grasp detection dataset featuring 1M samples, over 3M objects, and upwards of 10M grasping instructions. We utilize foundation models to create a large-scale scene corpus with corresponding images and grasp prompts. We approach the language-driven grasp detection task as a conditional generation problem. Drawing on the success of diffusion models in generative tasks and given that language plays a vital role in this task, we propose a new language-driven grasp detection method based on diffusion models. Our key contribution is the contrastive training objective, which explicitly contributes to the denoising process to detect the grasp pose given the language instructions. We illustrate that our approach is theoretically supportive. The intensive experiments show that our method outperforms state-of-the-art approaches and allows real-world robotic grasping. Finally, we demonstrate our large-scale dataset enables zero-short grasp detection and is a challenging benchmark for future work. Project website: https://airvlab.github.io/grasp-anything/
Paper Structure (26 sections, 1 theorem, 22 equations, 18 figures, 9 tables)

This paper contains 26 sections, 1 theorem, 22 equations, 18 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Grasp-Anything++ Dataset
  4. Prompting Procedure
  5. Image Synthesis and Grasp Annotation
  6. Data Statistics
  7. Language-driven Grasp Detection
  8. Constrastive Loss for Diffusion Model
  9. Language-driven Grasp Detection Network
  10. Experiments
  11. Language-driven Grasp Detection Results
  12. Zero-shot Grasp Detection
  13. Discussion
  14. Conclusion
  15. Theoretical Findings
  16. ...and 11 more sections

Key Result

Proposition 1

Suppose that $\tilde{\mathbf{x}}_0,\,\mathbf{x}_0$ and $\epsilon$ are independent, and that Then there exists $C>0$ such that: for arbitrary $\delta>0$, if $\mathcal{L}_{\rm{contrastive}}<\delta$, then

Figures (18)

  • Figure 1: We present a new dataset and method for language-driven grasp task.
  • Figure 2: Data statistics. We analyze object categories and the use of natural language in the Grasp-Anything++ dataset.
  • Figure 3: Failure image generation cases. Images generated by Stable Diffusion rombach2022high may exhibit hallucinatory artifacts such as sunglasses lacking a lens, scissors with an anomalous structure, and a spoon not resting properly on a table.
  • Figure 4: Language-drive Grasp Detection (LGD) network. We present the network architecture (left) and the proposed training objectives of the denoising process (right).
  • Figure 5: Language-driven grasp detection results visualization.
  • ...and 13 more figures

Theorems & Definitions (5)

  • Proposition 1
  • proof
  • Remark 1.1
  • Remark 1.2
  • proof