ObjectVLA: End-to-End Open-World Object Manipulation Without Demonstration

TL;DR

The paper tackles the open-world object generalization problem in end-to-end visuomotor policies by integrating vision-language priors with robot interaction data through localization-grounded reasoning.ObjectVLA uses a diffusion-based Vision-Language-Action model (DiVLA) co-trained on a hybrid dataset with a 10:1 robot-to-image-text data ratio and a 2B Qwen2-VL backbone to enable zero-shot manipulation of novel objects.Key findings show that ObjectVLA achieves 100% in-domain success and 64% success on 100 novel out-of-distribution objects, with additional demonstrated capabilities in bin-picking and other multi-step tasks, as well as rapid smartphone-based continual learning for new objects.The approach reduces reliance on large-scale human demonstrations and offers a practical, scalable path toward flexible real-world robotic manipulation in open-world settings.

Abstract

Imitation learning has proven to be highly effective in teaching robots dexterous manipulation skills. However, it typically relies on large amounts of human demonstration data, which limits its scalability and applicability in dynamic, real-world environments. One key challenge in this context is object generalization, where a robot trained to perform a task with one object, such as "hand over the apple," struggles to transfer its skills to a semantically similar but visually different object, such as "hand over the peach." This gap in generalization to new objects beyond those in the same category has yet to be adequately addressed in previous work on end-to-end visuomotor policy learning. In this paper, we present a simple yet effective approach for achieving object generalization through Vision-Language-Action (VLA) models, referred to as \textbf{ObjectVLA}. Our model enables robots to generalize learned skills to novel objects without requiring explicit human demonstrations for each new target object. By leveraging vision-language pair data, our method provides a lightweight and scalable way to inject knowledge about the target object, establishing an implicit link between the object and the desired action. We evaluate ObjectVLA on a real robotic platform, demonstrating its ability to generalize across 100 novel objects with a 64\% success rate in selecting objects not seen during training. Furthermore, we propose a more accessible method for enhancing object generalization in VLA models, using a smartphone to capture a few images and fine-tune the pre-trained model. These results highlight the effectiveness of our approach in enabling object-level generalization and reducing the need for extensive human demonstrations, paving the way for more flexible and scalable robotic learning systems.

Figures (7)

• Figure 1: Robot setup and examples for real-world manipulation tasks. We evaluate ObjectVLA with 4 skills on a Franka robot arm equipped with two external Zed cameras and a Realsense 435i wrist camera.
• Figure 2: Example of constructed image-text data.Left: Photo taken by the robot's camera. Right: Object captured with a smartphone.
• Figure 3: Example Objects Used in Experiments.Left: Objects present in the robot training data. Right: Examples of novel objects, not present in the robot data, but included in the image-text co-training dataset (see Appendix for a comprehensive list).
• Figure 4: Validation experiments on object generalization. Our method achieved the best performance in both the in-distribution test setup and under visual changes. Each object is evaluated across 4 trials. We report the number of objects that were correctly identified in all four trials.
• Figure 5: Experimental results for smartphone captured objects and trained by continual learning. We test two new objects. We took pictures of these two objects via smartphone and continually trained them on a pre-trained model. Each object was evaluated across 10 trials. We report the success rate for each object.