VLA^2: Empowering Vision-Language-Action Models with an Agentic Framework for Unseen Concept Manipulation

TL;DR

VLA^2 tackles the generalization gap of Vision-Language-Action models when encountering unseen concepts by introducing an agentic framework that augments a strong VLA backbone (OpenVLA) with external knowledge sources, task planning, and a verifier. The approach leverages web retrieval and object-detection grounding to convert novel objects and descriptions into in-distribution representations, enabling robust zero-shot manipulation in open-world simulations. Across LIBERO-based Easy/Medium/Hard benchmarks, VLA^2 significantly outperforms state-of-the-art VLAs, especially on hard OOD tasks (≈44.2% SR improvement) while preserving in-domain performance (≈20.2% average gain). The work demonstrates that system-level integration of planning, memory, and external modules can meaningfully extend the capabilities and reliability of vision-language robotic agents in unseen scenarios, with future work aimed at increasing autonomy and real-world validation.

Abstract

Current vision-language-action (VLA) models, pre-trained on large-scale robotic data, exhibit strong multi-task capabilities and generalize well to variations in visual and language instructions for manipulation. However, their success rate drops significantly when faced with object concepts outside the training data, such as unseen object descriptions and textures in the dataset. To address this, we propose a novel agentic framework, VLA^2, which leverages OpenVLA as the execution backbone and effectively leverages external modules such as web retrieval and object detection to provide visual and textual knowledge about target objects to the VLA. This approach mitigates generalization failure when handling out-of-distribution objects. Based on the LIBERO simulation environment, we introduced novel objects and object descriptions to construct a new evaluation benchmark with three difficulty levels to test the effectiveness of our method. Our framework successfully outperformed the current state-of-the-art models on our designed hard-level generalization benchmark. Compared to the standalone OpenVLA baseline, VLA^2 achieves a 44.2% improvement in the success rate in the hard-level benchmark and an average improvement of 20.2% in all customized environments without any performance degradation on in-domain tasks. Project website: https://vla-2.github.io.

Figures (7)

• Figure 1: Evaluation result on our custom Hard-level benchmark. In evaluation involving unseen concepts (i.e., object textures and language descriptions outside the dataset), our proposed framework surpasses other state-of-the-art models finetuned on the original LIBERO dataset. In contrast, the reproduced Agentic Robot framework Yang2025 using our model exhibits a significantly noticeable performance degradation in this task.
• Figure 2: Framework overview. The proposed framework comprises three components: A. preliminary processing, B. cognition and memory, and C. Judgment and execution. During task running, preliminary processing and cognition (except video object segmentation as VOS) are invoked only once at the start of each task.
• Figure 3: Vision framework. This figure illustrates the whole structure and contents within Vision.
• Figure 4: language framework. This figure illustrates the whole structure and contents within Language.
• Figure 5: Comparison between origin and new environments. In this figure, we illustrate the differences between the new and original environments. We present a single rendered scene to highlight the modified objects; the novel items appearing in the other scenes share the same appearance.