A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Clutter

TL;DR

A series of experiments indicate that the proposed jointly model vision, language and action with object-centric representation can achieve better task success rate by less times of motion under more flexible language instructions and is capable of generalizing better to scenarios with unseen objects and language instructions.

Abstract

We focus on the task of language-conditioned grasping in clutter, in which a robot is supposed to grasp the target object based on a language instruction. Previous works separately conduct visual grounding to localize the target object, and generate a grasp for that object. However, these works require object labels or visual attributes for grounding, which calls for handcrafted rules in planner and restricts the range of language instructions. In this paper, we propose to jointly model vision, language and action with object-centric representation. Our method is applicable under more flexible language instructions, and not limited by visual grounding error. Besides, by utilizing the powerful priors from the pre-trained multi-modal model and grasp model, sample efficiency is effectively improved and the sim2real problem is relived without additional data for transfer. A series of experiments carried out in simulation and real world indicate that our method can achieve better task success rate by less times of motion under more flexible language instructions. Moreover, our method is capable of generalizing better to scenarios with unseen objects and language instructions. Our code is available at https://github.com/xukechun/Vision-Language-Grasping

Figures (10)

• Figure 1: An example scenario of language-conditioned grasping in clutter. Our method jointly models vision, language and action with object-centric representation ( i.e. object bounding box), and conducts a sequence of grasps to grasp away obstacles, finally achieving the target object.
• Figure 2: System Overview. Given a language instruction, and object bounding boxes from detection module, our system pre-processes visual-language input by CLIP radford2021learning, and generates grasp poses by graspnet fang2020graspnet (see Sec. \ref{['method:preprocess']}). And we jointly model vision-language-action by a cross-attention transformer, of which outputs are fed into the policy and critic MLPs to generate logits and values of all grasps (see Sec. \ref{['method:joint']}).
• Figure 3: Test cases in simulation. The top two rows demonstrate the arrangements with seen objects, while the bottom row shows the arrangements with unseen objects. The target objects are labeled with stars.
• Figure 4: Compared training performance of our method and three methods.
• Figure 5: Ablation on all arrangements for CLIP (left) and RL vs. SL (right).