VLA Model-Expert Collaboration for Bi-directional Manipulation Learning

TL;DR

This work proposes a bi-directional expert-VLA collaboration framework that pairs Vision-Language-Action models with limited expert interventions to enhance manipulation generalization while lowering human workload. The approach supports two-way learning: the VLA model is fine-tuned on collaboration data, and human operators improve through interaction with the model. Across MetaWorld ML10/ML50 benchmarks, collaboration improves VLA success rates (e.g., up to 13.5% on MT50) and dramatically reduces human action steps (≈82% at N=4); preliminary SSVEP-based BCI experiments show substantially faster task completion when aided by the VLA model. The results demonstrate the practicality of bi-directional learning for robotic foundation models and point to online fine-tuning and real-world deployment as promising future directions.

Abstract

The emergence of vision-language-action (VLA) models has given rise to foundation models for robot manipulation. Although these models have achieved significant improvements, their generalization in multi-task manipulation remains limited. This study proposes a VLA model-expert collaboration framework that leverages a limited number of expert actions to enhance VLA model performance. This approach reduces expert workload relative to manual operation while simultaneously improving the reliability and generalization of VLA models. Furthermore, manipulation data collected during collaboration can further refine the VLA model, while human participants concurrently enhance their skills. This bi-directional learning loop boosts the overall performance of the collaboration system. Experimental results across various VLA models demonstrate the effectiveness of the proposed system in collaborative manipulation and learning, as evidenced by improved success rates across tasks. Additionally, validation using a brain-computer interface (BCI) indicates that the collaboration system enhances the efficiency of low-speed action systems by involving VLA model during manipulation. These promising results pave the way for advancing human-robot interaction in the era of foundation models for robotics. (Project website: https://aoqunjin.github.io/Expert-VLA/)

Figures (5)

• Figure 1: The proposed VLA model-expert collaboration system integrates a VLA model and expert interactions to enhance manipulation. The VLA model generates actions by processing task instructions as text tokens and environmental inputs as vision tokens. Meanwhile, the expert makes decisions at a lower frequency, assisting the VLA model. Expert-executed actions are collected to fine-tune the VLA model, improving system performance.
• Figure 2: Collaboration pipeline between VLA model and expert for manipulation and learning.
• Figure 3: Comparison of the baseline VLA model (Octo) and the VLA model after collaborative learning (tuning). The success rates of the fine-tuned VLA model—with and without the rule-based expert policy (V vs. V-R)—are presented at the task level (a) and at the average level (b) in the MT10 benchmark.
• Figure 4: Visualization of success rate and action steps executed by human expert over round in hard tasks (successful rate lower than average) under MT10.
• Figure 5: Application of the collaboration framework in SSVEP-based BCI: A comparison between pure SSVEP-based control and the collaboration between the VLA model and the BCI user. Although in some cases the policy of the human participant performs better than the VLA model (steps: 77 vs. 32), the collaboration system significantly improves time efficiency for a given task (time: 15s vs. 96s).