PGA: Personalizing Grasping Agents with Single Human-Robot Interaction

TL;DR

This work addresses the limitation of language-conditioned robotic grasping for personal objects by introducing GraspMine, a task scenario that enables grounding and grasping of user-specific items from a single human-robot interaction. It proposes the Personalized Grasping Agent (PGA), which builds a Reminiscence of unlabeled environment images, acquires personal object information through a two-step interaction, and propagates personal indicators to unlabeled objects via label propagation to personalize grounding. The method includes a Transformer-based Personalized Object Grounding Model and a practical grasping pipeline, validated both offline and online, with a real robot demonstration and public code. The GraspMine dataset and PGA demonstrate significant efficiency gains over fully supervised approaches while achieving competitive performance with orders of magnitude fewer annotations, enabling more natural and user-centric human-robot collaboration.

Abstract

Language-Conditioned Robotic Grasping (LCRG) aims to develop robots that comprehend and grasp objects based on natural language instructions. While the ability to understand personal objects like my wallet facilitates more natural interaction with human users, current LCRG systems only allow generic language instructions, e.g., the black-colored wallet next to the laptop. To this end, we introduce a task scenario GraspMine alongside a novel dataset aimed at pinpointing and grasping personal objects given personal indicators via learning from a single human-robot interaction, rather than a large labeled dataset. Our proposed method, Personalized Grasping Agent (PGA), addresses GraspMine by leveraging the unlabeled image data of the user's environment, called Reminiscence. Specifically, PGA acquires personal object information by a user presenting a personal object with its associated indicator, followed by PGA inspecting the object by rotating it. Based on the acquired information, PGA pseudo-labels objects in the Reminiscence by our proposed label propagation algorithm. Harnessing the information acquired from the interactions and the pseudo-labeled objects in the Reminiscence, PGA adapts the object grounding model to grasp personal objects. This results in significant efficiency while previous LCRG systems rely on resource-intensive human annotations -- necessitating hundreds of labeled data to learn my wallet. Moreover, PGA outperforms baseline methods across all metrics and even shows comparable performance compared to the fully-supervised method, which learns from 9k annotated data samples. We further validate PGA's real-world applicability by employing a physical robot to execute GrsapMine. Code and data are publicly available at https://github.com/JHKim-snu/PGA.

Figures (4)

• Figure 1: Personalizing grasping agents with single human-robot interaction. Upon the user's introduction of a personal object, it retrieves the identical objects from its visual reminiscence. Leveraging the retrieved objects, the robot subsequently engages in an integrated learning process of the personal object. The Personalized Grasping Agent (PGA) can finally comprehend and grasp the personal object.
• Figure 2: Overview of Personalized Grasping Agent (PGA). (a) Initially, PGA gathers a collection of raw images, termed as Reminiscence, from the user's environment. With the personal indicator provided by the user from (b), unlabeled objects in the Reminiscence are pseudo-labeled via (c) Propagation through Reminiscence. It's vital to note that certain objects, particularly those not introduced by the user (e.g., non-personal objects), remain unlabeled by our algorithm. Ultimately, PGA employs all the object nodes with labels (colored nodes) to train the Personalized Object Grounding Model.
• Figure 3: Impact of Reminiscence size. PGA's offline scores based on the number of raw images utilized from Reminiscence, from '0' to '400'.
• Figure 4: Qualitative Analysis. The top row showcases examples of objects from each phase. In the bottom row, personalized object grounding results by PGA are depicted in blue boxes and those by PassivePGA model in red, set alongside the ground truth in black boxes. Within the Propagation through Reminiscence on the top row, solid lined boxes are the pseudo-labeled objects from the Reminiscence and dotted boxes denote objects that were NOT pseudo-labeled according to indicators in the respective models.