Unveiling the Potential of Vision-Language-Action Models with Open-Ended Multimodal Instructions

TL;DR

OE-VLA extends vision-language-action models to accept open-ended multimodal instructions beyond pure language, enabling robots to follow prompts that include images, text overlays, and video demonstrations. The approach combines a SigLIP ViT-based vision encoder, a Qwen-1.5b language backbone, and a discretized 256-bin action tokenizer within an interleaved multi-image framework, trained via a two-stage curriculum (multi-image grounding then open-ended instruction tuning). It introduces two benchmarks, OE-CALVIN_base and OE-CALVIN_hard, to evaluate open-ended multimodal prompting, and shows that OE-VLA matches linguistic VLA performance while delivering strong results on diverse multimodal prompts, including VOS, OIF, VGR, and VDL. The work also presents data-construction and training-data recipes that repurpose existing robotic datasets into multimodal instruction formats and demonstrates the benefits of a staged training pipeline for cross-modal grounding and instruction understanding. Overall, the method broadens the applicability of VLA models to real-world human–robot interaction scenarios where prompts come from images, handwritten text, and demonstrations, not just language.

Abstract

Vision-Language-Action (VLA) models have recently become highly prominent in the field of robotics. Leveraging vision-language foundation models trained on large-scale internet data, the VLA model can generate robotic actions directly from visual observations and human instructions through a single end-to-end neural network. Despite their effectiveness, current VLA models usually accept only one form of human prompting, language instructions, which may constrain their applicability in open-ended human-robot interactions. For example, a user might expect the robot to retrieve an object shown in an image, follow an instruction written on the whiteboard, or imitate a behavior demonstrated in a video, rather than relying solely on language-based descriptions. To address this gap, we introduce OE-VLA, which explores the potential of VLA models for open-ended multimodal instructions. Extensive results demonstrate that our OE-VLA not only achieves comparable performance to traditional VLA models with linguistic input but also delivers impressive results across four additional categories of open-ended tasks. The proposed methodology could significantly expand the applications of VLA models across various everyday scenarios and facilitate human-robot interaction.

Figures (6)

• Figure 1: The introduced OE-VLA with four open-ended robot manipulation tasks.
• Figure 2: The comprehensive architecture of our OE-VLA model.
• Figure 3: The data construction method for transforming traditional dataset into the target dataset with open-ended instructions.
• Figure 4: The introduced two new benchmarks with diverse open-ended task instructions.
• Figure 5: Performance of our method on the OE-CALVIN$_{base}$ benchmark. The upper row presents the results for the ABC$\to$D split, while the lower row presents the results for the ABCD$\to$D split.