CrayonRobo: Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation

TL;DR

CrayonRobo tackles the challenge of specifying robotic manipulation goals by using crayon-style visual prompts over key-frame RGB images to express both low-level actions and high-level planning. It trains a Vision-Language-Action model, leveraging a CLIP-based visual encoder and a frozen LLaMA-based text module with LoRA adapters, to predict 6-DoF end-effector poses in $SE(3)$ and subsequent movement directions from multi-modal prompts, with losses $L_T$, $L_O$, and $L_P$ guiding alignment, orthogonality, and 2D-3D consistency. The approach enables sequential execution of key-frame steps to accomplish long-horizon tasks, demonstrates robust zero-shot performance on unseen tasks, and shows resilience to prompt noise in both simulation and real-world experiments. The work delivers concrete benefits for interpretable goal specification and robust manipulation, while noting limitations in obstacle avoidance and suggesting future integration with collision-free planners for safer deployment.

Abstract

In robotic, task goals can be conveyed through various modalities, such as language, goal images, and goal videos. However, natural language can be ambiguous, while images or videos may offer overly detailed specifications. To tackle these challenges, we introduce CrayonRobo that leverages comprehensive multi-modal prompts that explicitly convey both low-level actions and high-level planning in a simple manner. Specifically, for each key-frame in the task sequence, our method allows for manual or automatic generation of simple and expressive 2D visual prompts overlaid on RGB images. These prompts represent the required task goals, such as the end-effector pose and the desired movement direction after contact. We develop a training strategy that enables the model to interpret these visual-language prompts and predict the corresponding contact poses and movement directions in SE(3) space. Furthermore, by sequentially executing all key-frame steps, the model can complete long-horizon tasks. This approach not only helps the model explicitly understand the task objectives but also enhances its robustness on unseen tasks by providing easily interpretable prompts. We evaluate our method in both simulated and real-world environments, demonstrating its robust manipulation capabilities.

Figures (6)

• Figure 1: (a) shows our expression of different color prompts. (b) shows that we utilize a sequence of images with crayon visual prompts to express the key-frame steps, with each step illustrating the required low-level task goals, i.e., t1-pick, t2-place. Based on the input goal prompt, the model determines the 6 DoF contact pose, enabling it to interact with the object as required. When a yellow prompt is presented, the model also predicts 3D movement directions, guiding the movement after contact, e.g., picking upward in t1. For simple steps, such as t2-place, there is no need to present the post-contact moving direction. By sequentially executing each step in the key-frame sequence, the overall task is completed. (c) shows its differences with RT-trajectory brohan2023rt
• Figure 2: We design training pairs that convey varying levels of information to enable the model to comprehend each type of prompt and introduce loss functions to guide it in predicting accurate poses.
• Figure 3: Illustration of model inference with input generated in different ways.
• Figure 4: Visualization results in SAPIEN simulator and real world.
• Figure 5: Robustness analysis on the noise in prompts.