Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Parameter-Efficient Tuning Framework for Language-guided Object Grounding and Robot Grasping

Houjian Yu, Mingen Li, Alireza Rezazadeh, Yang Yang, Changhyun Choi

TL;DR

The paper tackles language-guided object grounding and robot grasping under open-vocabulary settings by introducing a CLIP-based parameter-efficient tuning framework. It deploys a bidirectional vision-language adapter for cross-modal early fusion and a depth fusion branch to incorporate geometric cues, all atop a frozen CLIP backbone, followed by a multimodal transformer decoder to support RES, RGS, and RGA. The approach yields competitive results with far fewer tunable parameters than full-model fine-tuning, and depth information notably improves grasp predictions and spatial reasoning, both in simulation and real-robot tests. This framework enables efficient customization and deployment of language-guided grounding and grasping systems on robots, with strong potential for handling complex linguistic descriptions and spatial relationships in real-world environments.

Abstract

The language-guided robot grasping task requires a robot agent to integrate multimodal information from both visual and linguistic inputs to predict actions for target-driven grasping. While recent approaches utilizing Multimodal Large Language Models (MLLMs) have shown promising results, their extensive computation and data demands limit the feasibility of local deployment and customization. To address this, we propose a novel CLIP-based multimodal parameter-efficient tuning (PET) framework designed for three language-guided object grounding and grasping tasks: (1) Referring Expression Segmentation (RES), (2) Referring Grasp Synthesis (RGS), and (3) Referring Grasp Affordance (RGA). Our approach introduces two key innovations: a bi-directional vision-language adapter that aligns multimodal inputs for pixel-level language understanding and a depth fusion branch that incorporates geometric cues to facilitate robot grasping predictions. Experiment results demonstrate superior performance in the RES object grounding task compared with existing CLIP-based full-model tuning or PET approaches. In the RGS and RGA tasks, our model not only effectively interprets object attributes based on simple language descriptions but also shows strong potential for comprehending complex spatial reasoning scenarios, such as multiple identical objects present in the workspace. Project page: https://z.umn.edu/etog-etrg

A Parameter-Efficient Tuning Framework for Language-guided Object Grounding and Robot Grasping

TL;DR

The paper tackles language-guided object grounding and robot grasping under open-vocabulary settings by introducing a CLIP-based parameter-efficient tuning framework. It deploys a bidirectional vision-language adapter for cross-modal early fusion and a depth fusion branch to incorporate geometric cues, all atop a frozen CLIP backbone, followed by a multimodal transformer decoder to support RES, RGS, and RGA. The approach yields competitive results with far fewer tunable parameters than full-model fine-tuning, and depth information notably improves grasp predictions and spatial reasoning, both in simulation and real-robot tests. This framework enables efficient customization and deployment of language-guided grounding and grasping systems on robots, with strong potential for handling complex linguistic descriptions and spatial relationships in real-world environments.

Abstract

The language-guided robot grasping task requires a robot agent to integrate multimodal information from both visual and linguistic inputs to predict actions for target-driven grasping. While recent approaches utilizing Multimodal Large Language Models (MLLMs) have shown promising results, their extensive computation and data demands limit the feasibility of local deployment and customization. To address this, we propose a novel CLIP-based multimodal parameter-efficient tuning (PET) framework designed for three language-guided object grounding and grasping tasks: (1) Referring Expression Segmentation (RES), (2) Referring Grasp Synthesis (RGS), and (3) Referring Grasp Affordance (RGA). Our approach introduces two key innovations: a bi-directional vision-language adapter that aligns multimodal inputs for pixel-level language understanding and a depth fusion branch that incorporates geometric cues to facilitate robot grasping predictions. Experiment results demonstrate superior performance in the RES object grounding task compared with existing CLIP-based full-model tuning or PET approaches. In the RGS and RGA tasks, our model not only effectively interprets object attributes based on simple language descriptions but also shows strong potential for comprehending complex spatial reasoning scenarios, such as multiple identical objects present in the workspace. Project page: https://z.umn.edu/etog-etrg
Paper Structure (15 sections, 1 equation, 4 figures, 5 tables)

This paper contains 15 sections, 1 equation, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Language-guided Object Grounding
  4. Language-guided Robot Grasping
  5. Parameter-Efficient Tuning
  6. Methods
  7. Problem Formulation
  8. Multimodal Feature Early Fusion
  9. Multimodal Transformer Decoder
  10. Projectors and Training Objectives
  11. Experiments
  12. Referring Expression Segmentation for Object Grounding
  13. Referring Grasp Synthesis with Depth Fusion
  14. Referring Grasp Affordance for Spatial Reasoning
  15. Conclusion

Figures (4)

  • Figure 1: Efficient-Tuning pipeline for language-guided object grounding and robot grasping tasks. We propose Efficient-Tuning Object Grounding (ETOG) for RES ask, Efficient-Tuning Robot Grasping type-A (ETRG-A) for RGS task, and type-B (ETRG-B) for RGA task. Our framework with minor modifications is able to solve the three tasks. Zoom-in for more details.
  • Figure 2: Our model takes visual, language and depth (optional, depending on tasks) as inputs and outputs the task predictions. Our VL-adapter fuses the multimodal features extracted from CLIP at different stages and previous VL-adapter layers. The Pixel-Sentence Fusion module combines multi-scale visual features with a sentence-level global representation, while the Pixel-Words Fusion module focuses on word-level token understanding and eventually generates the segmentation masks.
  • Figure 3: Qualitative results on model prediction and feature visualization.Zoom-in for more details.
  • Figure 4: Qualitative results on grasp affordance prediction with ETRG-B. The green bounding boxes highlight the language-referred target. The language expressions used are: (1) "The Mustard Bottle that is to the middle left of the workspace", (2) "Yellow Mustard Bottle that is to the upper right of the workspace", (3) "lower left Mustard Bottle" and (4) "Yellow Mustard Bottle that is to the upper right of the apple".