UniDiffGrasp: A Unified Framework Integrating VLM Reasoning and VLM-Guided Part Diffusion for Open-Vocabulary Constrained Grasping with Dual Arms

TL;DR

UniDiffGrasp tackles open-vocabulary, task-oriented grasping with dual arms by marrying Vision-Language Model reasoning and CGDF-based constrained grasp diffusion. It grounds VLM-derived targets into geometric constraints using GroundedSAM and VLPart, then uses Part-Guided Diffusion to generate robust 6-DoF grasps conditioned on both the global object and the targeted region. For dual-arm tasks, it defines two target regions and independently generates grasps per arm, followed by energy-based filtering, collision checks, and force-closure validation to select a stable pair. Real-world experiments on a Baxter robot demonstrate state-of-the-art performance, with GSR of 0.876 for single-arm and 0.767 for dual-arm grasps and CFRs up to 0.900 and 0.850, respectively, highlighting effective open-vocabulary grasping and coordinated dual-arm manipulation in complex settings.

Abstract

Open-vocabulary, task-oriented grasping of specific functional parts, particularly with dual arms, remains a key challenge, as current Vision-Language Models (VLMs), while enhancing task understanding, often struggle with precise grasp generation within defined constraints and effective dual-arm coordination. We innovatively propose UniDiffGrasp, a unified framework integrating VLM reasoning with guided part diffusion to address these limitations. UniDiffGrasp leverages a VLM to interpret user input and identify semantic targets (object, part(s), mode), which are then grounded via open-vocabulary segmentation. Critically, the identified parts directly provide geometric constraints for a Constrained Grasp Diffusion Field (CGDF) using its Part-Guided Diffusion, enabling efficient, high-quality 6-DoF grasps without retraining. For dual-arm tasks, UniDiffGrasp defines distinct target regions, applies part-guided diffusion per arm, and selects stable cooperative grasps. Through extensive real-world deployment, UniDiffGrasp achieves grasp success rates of 0.876 in single-arm and 0.767 in dual-arm scenarios, significantly surpassing existing state-of-the-art methods, demonstrating its capability to enable precise and coordinated open-vocabulary grasping in complex real-world scenarios.

Figures (5)

• Figure 1: UniDiffGrasp: Open-Vocabulary Constrained Grasping Example and Performance Comparison. UniDiffGrasp innovatively applies a VLM-guided part diffusion strategy, achieving state-of-the-art open-vocabulary part grasping and uniquely extending this precision to complex dual-arm cooperation. Its unified end-to-end framework, integrating single and dual-arm capabilities, demonstrates exceptional real-world performance and strong generalization.
• Figure 2: UniDiffGrasp Framework: Integrating VLM Reasoning, Multi-Stage Segmentation, and Part-Guided Diffusion for Grasp Generation. User input (e.g., 'I'd like to stew some soup') and visual context are first processed by a VLM (GPT-4o) to determine the target object ('pot'), functional part ('handle'), and operation mode ('dual-arm'). Multi-stage segmentation—Grounded SAM for object localization and VLPart for precise part identification—then grounds these semantic targets into geometric constraints. Finally, these constraints guide the Part-Guided Diffusion strategy, employing the dual-arm coordination method (Sec. III-C) to define distinct target regions (e.g., for each handle of the pot) and select stable cooperative grasps.
• Figure 3: Principle of Dual-Arm Grasp Generation in UniDiffGrasp using Coordinated Part-Guided Diffusion. We illustrate how UniDiffGrasp generates dual-arm grasps, taking a pot with two handles as an example.
• Figure 4: Visualization of task-oriented grasping of functional parts by single-arm
• Figure 5: Visualization of collaborative dual-arm grasping for large object manipulation