PROGrasp: Pragmatic Human-Robot Communication for Object Grasping

TL;DR

PROGrasp introduces Pragmatic-IOG, a reasoning framework where a robot infers a target object from intention-oriented natural language through interactive dialogue. It integrates four modules—visual grounding, question generation, answer interpretation, and object grasping—with a pragmatic inference step that records context from both vision and dialogue, weighted by a rationality parameter $\lambda$. The accompanying IM-Dial dataset enables training these modules end-to-end and evaluating both offline object discovery and online grasping with a real robot, demonstrating substantial gains over baselines and showing process efficiency through fewer interactions. The work advances human-centric robotics by embedding pragmatic, context-aware reasoning into grounding and manipulation tasks, and it provides a foundation for future pragmatic robotic systems and datasets.

Abstract

Interactive Object Grasping (IOG) is the task of identifying and grasping the desired object via human-robot natural language interaction. Current IOG systems assume that a human user initially specifies the target object's category (e.g., bottle). Inspired by pragmatics, where humans often convey their intentions by relying on context to achieve goals, we introduce a new IOG task, Pragmatic-IOG, and the corresponding dataset, Intention-oriented Multi-modal Dialogue (IM-Dial). In our proposed task scenario, an intention-oriented utterance (e.g., "I am thirsty") is initially given to the robot. The robot should then identify the target object by interacting with a human user. Based on the task setup, we propose a new robotic system that can interpret the user's intention and pick up the target object, Pragmatic Object Grasping (PROGrasp). PROGrasp performs Pragmatic-IOG by incorporating modules for visual grounding, question asking, object grasping, and most importantly, answer interpretation for pragmatic inference. Experimental results show that PROGrasp is effective in offline (i.e., target object discovery) and online (i.e., IOG with a physical robot arm) settings. Code and data are available at https://github.com/gicheonkang/prograsp.

Figures (7)

• Figure 1: Overview of interactive object grasping with intention-oriented utterance. The initial utterance does not contain the target object's category.
• Figure 2: Illustration of the inference step in PROGrasp when $\bm{T=1}$. VG first performs object grounding using the dialogue history. Q-gen then selects the object candidate to ask and generates a question. After obtaining the response from the user, VG cooperates with A-int to determine the target object region. The object grasping module finally grasps the object by computing the 3D coordinates of the target object.
• Figure 3: The 86 categories of everyday objects used in the experiments.
• Figure 4: Validation scores adjusting the hyperparameters, $\lambda$ and $T$.
• Figure 5: Visualization of PROGrasp's target object recovery.