Object-Centric Instruction Augmentation for Robotic Manipulation

TL;DR

This work tackles the challenge of teaching robots to manipulate objects when language alone is insufficient to convey where objects are. It introduces Object-Centric Instruction Augmentation (OCI), which uses a position-aware Multimodal Large Language Model (MLLM) to augment natural-language instructions with absolute bounding boxes and relative spatial relations to the robot, paired with a Feature Reuse Mechanism (FRM) that injects MLLM embeddings into the policy network through multi-scale cross-attention. The MLLM is fine-tuned on detection and instruction datasets to rewrite commands into augmented forms such as "+object positions+" encoded as $[x_{min}, y_{min}, x_{max}, y_{max}]$ and eight relative directions, while the policy learning backbone remains lightweight and efficient. Across simulation (Franka Kitchen) and real-robot experiments, OCI consistently outperforms baselines like R3M and BLIP-2, and ablations confirm the critical roles of both absolute/relative position cues and FRM in improving generalization and success rates. Overall, by delegating position localization to language-augmented instructions and reusing MLLM knowledge, the approach enhances sample efficiency and robustness for versatile robotic manipulation in unseen environments.

Abstract

Humans interpret scenes by recognizing both the identities and positions of objects in their observations. For a robot to perform tasks such as \enquote{pick and place}, understanding both what the objects are and where they are located is crucial. While the former has been extensively discussed in the literature that uses the large language model to enrich the text descriptions, the latter remains underexplored. In this work, we introduce the \textit{Object-Centric Instruction Augmentation (OCI)} framework to augment highly semantic and information-dense language instruction with position cues. We utilize a Multi-modal Large Language Model (MLLM) to weave knowledge of object locations into natural language instruction, thus aiding the policy network in mastering actions for versatile manipulation. Additionally, we present a feature reuse mechanism to integrate the vision-language features from off-the-shelf pre-trained MLLM into policy networks. Through a series of simulated and real-world robotic tasks, we demonstrate that robotic manipulator imitation policies trained with our enhanced instructions outperform those relying solely on traditional language instructions.

Figures (7)

• Figure 1: Two examples of object-centric instruction augmentation for simulation and real robots, respectively. Given an initial instruction from the left figure, we augment them by providing the object's absolute position and relative position to the robotics and obtain the action eventually.
• Figure 2: An overview of the training process in OCI. We first fine-tune a MLLM with general detection datasets and our collected datasets. The fine-tuned MLLM enables automatically augmenting instruction object-centric information. Subsequently, this is integrated with a policy network to develop a model capable of generating specific actions.
• Figure 3: This figure illustrates how we connect the policy network with MLLM. During the fine-tuning phase, MLLM is only utilized at the initial time step $t_{1}$, and its parameters are frozen.
• Figure 4: The experimental results on Franka Kitchen. On all sub-tasks, our proposed OCI beats existing approaches, where our methods lead for a large margin on some tasks.
• Figure 5: The example of Franka Kitchen for five tasks on two camera views.