Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting

TL;DR

This work tackles the challenge of personal object manipulation with Vision-Language-Action models by introducing Visual Attentive Prompting (VAP), a training-free perceptual adapter that grounds a user-specific object from a few reference images and injects a visual prompt to guide a frozen VLA. By decomposing personalization into grounding and prompting, VAP leverages a non-parametric memory and real-time tracking to produce a prompted observation and rewritten instruction, enabling immediate use of existing VLAs for instance-level control. Across two simulation benchmarks and a real-world tabletop setup, VAP substantially outperforms generic and token-based baselines in both success rate and correct-object manipulation, demonstrating a practical path toward personalized embodied AI without per-object fine-tuning. The results also highlight limitations related to segmentation quality and cross-view consistency, pointing to future work on stronger 3D grounding, affordance-aware personalization, and multi-user collaboration in home and office settings.

Abstract

While Vision-Language-Action (VLA) models generalize well to generic instructions, they struggle with personalized commands such as "bring my cup", where the robot must act on one specific instance among visually similar objects. We study this setting of manipulating personal objects, in which a VLA must identify and control a user-specific object unseen during training using only a few reference images. To address this challenge, we propose Visual Attentive Prompting (VAP), a simple-yet-effective training-free perceptual adapter that equips frozen VLAs with top-down selective attention. VAP treats the reference images as a non-parametric visual memory, grounds the personal object in the scene through open-vocabulary detection and embedding-based matching, and then injects this grounding as a visual prompt by highlighting the object and rewriting the instruction. We construct two simulation benchmarks, Personalized-SIMPLER and Personalized-VLABench, and a real-world tabletop benchmark to evaluate personalized manipulation across multiple robots and tasks. Experiments show that VAP consistently outperforms generic policies and token-learning baselines in both success rate and correct-object manipulation, helping to bridge the gap between semantic understanding and instance-level control.

Figures (20)

• Figure 1: Manipulating personal objects with VLA. Existing vision-language-action (VLA) models cannot handle personal objects such as <my cup>, because they can only interpret generic, language-expressible semantics. We address this limitation with a simple framework, Visual Attentive Prompting (VAP). It first grounds the user-specific object in the scene by matching it against the memory and then uses visual prompting to guide the VLA. This pipeline enables existing VLA models to manipulate personal objects without any additional training.
• Figure 2: Overview of Evaluation Benchmarks. We evaluate our method across simulation and real-world tasks. Top (Simulation): We construct Personalized-SIMPLER (left/middle) and Personalized-VLABench (right) by repopulating existing environments with user-specific assets. Bottom (Real-world): We conduct physical experiments on a SO-101 arm using 8 diverse object categories, covering both selection and pick-and-place tasks. In all scenarios, the agent must identify a specific target instance among visually similar distractors, requiring precise instance-level grounding beyond generic category recognition.
• Figure 3: VAP builds a memory from a few reference images and grounds the target object with frozen detection and segmentation modules. It then highlights that object and rewrites the instruction for a frozen VLA, tracking it over subsequent frames and enabling training-free personalized actions.
• Figure 4: Overview of our real-world experimental setup.
• Figure 5: Performance on real-world tasks, averaged over 20 trials. We report Success Rate (SR) for all tasks and Correct Movement Ratio (CMR) for pick-and-place tasks. The CMR metric is not applicable for pointing tasks. VAP demonstrates a consistently high success rate across all scenarios, consistently outperforming the baseline.