Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ORACLE-Grasp: Zero-Shot Task-Oriented Robotic Grasping using Large Multimodal Models

Avihai Giuili, Rotem Atari, Avishai Sintov

TL;DR

ORACLE-Grasp tackles the challenge of grasping unknown objects in unstructured environments by combining semantic reasoning from large multimodal models with lightweight vision cues. It reframes grasping as an iterative, prompt-driven decision process using dual prompts to extract scene context and candidate grasp regions, followed by depth-based refinement and early stopping to ensure feasibility. The approach achieves competitive accuracy and high real-world success (88% on 20 objects) without any task-specific training data, highlighting the potential of language-driven reasoning to enable autonomous grasping. The work also discusses limitations such as inference cost and single-view perception, with clear avenues for future improvements.

Abstract

Grasping unknown objects in unstructured environments remains a fundamental challenge in robotics, requiring both semantic understanding and spatial reasoning. Existing methods often rely on dense training datasets or explicit geometric modeling, limiting their scalability to real-world tasks. Recent advances in Large Multimodal Models (LMMs) offer new possibilities for integrating vision and language understanding, but their application to autonomous robotic grasping remains largely unexplored. We present ORACLE-Grasp, a zero-shot framework that leverages LMMs as semantic oracles to guide grasp selection without requiring additional training or human input. The system formulates grasp prediction as a structured, iterative decision process, using dual-prompt tool calling to first extract high-level object context and then select task-relevant grasp regions. By discretizing the image space and reasoning over candidate areas, ORACLE-Grasp mitigates the spatial imprecision common in LMMs and produces human-like, task-driven grasp suggestions. Early stopping and depth-based refinement steps further enhance efficiency and physical grasp reliability. Experiments demonstrate that the predicted grasps achieve low positional and orientation errors relative to human-annotated ground truth and lead to high success rates in real-world pick up tasks. These results highlight the potential of combining language-driven reasoning with lightweight vision techniques to enable robust, autonomous grasping without task-specific datasets or retraining.

ORACLE-Grasp: Zero-Shot Task-Oriented Robotic Grasping using Large Multimodal Models

TL;DR

ORACLE-Grasp tackles the challenge of grasping unknown objects in unstructured environments by combining semantic reasoning from large multimodal models with lightweight vision cues. It reframes grasping as an iterative, prompt-driven decision process using dual prompts to extract scene context and candidate grasp regions, followed by depth-based refinement and early stopping to ensure feasibility. The approach achieves competitive accuracy and high real-world success (88% on 20 objects) without any task-specific training data, highlighting the potential of language-driven reasoning to enable autonomous grasping. The work also discusses limitations such as inference cost and single-view perception, with clear avenues for future improvements.

Abstract

Grasping unknown objects in unstructured environments remains a fundamental challenge in robotics, requiring both semantic understanding and spatial reasoning. Existing methods often rely on dense training datasets or explicit geometric modeling, limiting their scalability to real-world tasks. Recent advances in Large Multimodal Models (LMMs) offer new possibilities for integrating vision and language understanding, but their application to autonomous robotic grasping remains largely unexplored. We present ORACLE-Grasp, a zero-shot framework that leverages LMMs as semantic oracles to guide grasp selection without requiring additional training or human input. The system formulates grasp prediction as a structured, iterative decision process, using dual-prompt tool calling to first extract high-level object context and then select task-relevant grasp regions. By discretizing the image space and reasoning over candidate areas, ORACLE-Grasp mitigates the spatial imprecision common in LMMs and produces human-like, task-driven grasp suggestions. Early stopping and depth-based refinement steps further enhance efficiency and physical grasp reliability. Experiments demonstrate that the predicted grasps achieve low positional and orientation errors relative to human-annotated ground truth and lead to high success rates in real-world pick up tasks. These results highlight the potential of combining language-driven reasoning with lightweight vision techniques to enable robust, autonomous grasping without task-specific datasets or retraining.
Paper Structure (8 sections, 5 figures, 2 tables)

This paper contains 8 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Experiments
  4. Test-sets
  5. Evaluation metric
  6. Grasp error evaluation
  7. Robot grasping experiments
  8. Conclusions

Figures (5)

  • Figure 1: Overview of the ORACLE-Grasp framework. An RGB-D camera captures the scene. An LMM uses dual prompts over a grid to identify a graspable region, which is then refined with depth data for precise grasp prediction and executed by a robot.
  • Figure 2: Predicted grasps on three RGB images from test-set I, showing position (blue dot) and orientation (blue line). Objects include, from left to right: a vacuum cleaner, an AI-generated drill and a pencil. Red cells indicate regions selected by the LMM as potential grasping areas.
  • Figure 3: Predicted grasps (blue) versus human annotations (red) on RGB images from test-set I. Objects shown with their respective NRMSE: (a) Mug – 0.0385, (b) Saw – 0.027, (c) Adjustable wrench – 0.026, (d) Frying pan – 0.020, (e) Robot arm – 0.086, (f) Dumbbell – 0.074, (g) Hair dryer – 0.018, and (h) Suitcase – 0.046.
  • Figure 4: Position NRMSE and computation runtime with regard to the number of GRP queries $K$ exerted in O-Grasp.
  • Figure 5: Real-world objects from test-set II being grasped by a robot arm using predictions from O-Grasp. For each object, the figure shows a snapshot of the robot executing the grasp and the corresponding RGB image captured by the on-board camera. The demonstrated objects include: (a) cardboard box, (b) coffee bag, (c) six-arm soldering holder, (d) toothpaste tube, (e) dustpan, and (f) Plastic cylinder. In some cases, the initial predicted grasp (in green) was refined using depth data (final grasp shown in red), as described in Section \ref{['sec:adjust']}. The green circles mark the clearance region with radius $r(z_{\bar{\mathbf{p}}})$.