Attribute-Based Robotic Grasping with Data-Efficient Adaptation

TL;DR

This work tackles target-driven robotic grasping in clutter by using object attributes to generalize to novel objects. It introduces an end-to-end multimodal encoder-decoder that fuses visual observations with attribute text via gated-attention and learns a joint embedding space through object persistence to predict instance grasping affordances. To address domain shift and data scarcity, two data-efficient adaptation methods are proposed: adversarial adaptation to learn domain-invariant features and one-grasp adaptation to fine-tune with a single grasp trial, yielding substantial gains in both simulation and real-world tests. The approach demonstrates strong performance on unknown objects, with over 81% adapted instance grasping success in real-world novel objects and competitive results against baselines and a foundation model, highlighting practical potential for rapid deployment in cluttered environments.

Abstract

Robotic grasping is one of the most fundamental robotic manipulation tasks and has been the subject of extensive research. However, swiftly teaching a robot to grasp a novel target object in clutter remains challenging. This paper attempts to address the challenge by leveraging object attributes that facilitate recognition, grasping, and rapid adaptation to new domains. In this work, we present an end-to-end encoder-decoder network to learn attribute-based robotic grasping with data-efficient adaptation capability. We first pre-train the end-to-end model with a variety of basic objects to learn generic attribute representation for recognition and grasping. Our approach fuses the embeddings of a workspace image and a query text using a gated-attention mechanism and learns to predict instance grasping affordances. To train the joint embedding space of visual and textual attributes, the robot utilizes object persistence before and after grasping. Our model is self-supervised in a simulation that only uses basic objects of various colors and shapes but generalizes to novel objects in new environments. To further facilitate generalization, we propose two adaptation methods, adversarial adaption and one-grasp adaptation. Adversarial adaptation regulates the image encoder using augmented data of unlabeled images, whereas one-grasp adaptation updates the overall end-to-end model using augmented data from one grasp trial. Both adaptation methods are data-efficient and considerably improve instance grasping performance. Experimental results in both simulation and the real world demonstrate that our approach achieves over 81% instance grasping success rate on unknown objects, which outperforms several baselines by large margins.

Figures (14)

• Figure 1: Attribute-based instance grasping. Various objects of generic attributes are placed in the workspace, and we propose to grasp a target object by describing its attributes, e.g., "Please give me the apple, a red sphere.".
• Figure 2: Overview of affordances and attribute learning. The workspace image and query text are encoded separately and fused using gated-attention. The fusion matrix $F_{\text{att}}$ is rotated by $N$ orientations for different grasping angles and then fed into the grasping affordances decoder. The decoder learns to predict pixel-wise scores of target grasping success, and we run the $\epsilon$-greedy grasping policy and obtain the image $v_{\text{post}}$ after grasping. By utilizing the equation of object persistence before and after grasping, we learn a metric space where multimodal embedding vectors corresponding to similar attributes are encouraged to be closer. Note we denote the combination of $\phi_{v, \text{spa}}$ and GAP as $\phi_{v, \text{vec}}$, which encodes images to vectors.
• Figure 3: Multimodal feature space supervised by the equation of object persistence (\ref{['eq:metric']}). The image and text encoders are trained to produce consistent embeddings, where feature vectors corresponding to similar attributes are encouraged to be closer.
• Figure 4: Examples of basic objects. Synthetic objects of various colors and shapes are used for learning object attributes and grasping affordances. To ensure shape attribute learning, we include objects having random textures.
• Figure 5: Testing objects in simulation and the real world. We use the testing objects that share similar attributes with the training objects. See Appendix for more details.