MoS-VLA: A Vision-Language-Action Model with One-Shot Skill Adaptation

TL;DR

MoS-VLA addresses the brittleness of vision-language-action robotics models by learning a structured space of policies as a linear combination of basis functions trained across diverse labs. Adaptation to a new task or environment is achieved in a gradient-free manner by solving a small $L^1$ optimization problem to obtain coefficients for the basis functions, requiring only a single expert demonstration. Empirically, MoS-VLA reduces $L^1$ error on both in-distribution and unseen datasets and dramatically improves success rates on simulated and real-robot tasks compared to pretrained OpenVLA, highlighting the practical impact of gradient-free, in-context skill adaptation. This work opens a path toward scalable, rapid skill acquisition in robot foundation models without costly fine-tuning.

Abstract

Vision-Language-Action (VLA) models trained on large robot datasets promise general-purpose, robust control across diverse domains and embodiments. However, existing approaches often fail out-of-the-box when deployed in novel environments, embodiments, or tasks. We introduce Mixture of Skills VLA (MoS-VLA), a framework that represents robot manipulation policies as linear combinations of a finite set of learned basis functions. During pretraining, MoS-VLA jointly learns these basis functions across datasets from the Open X-Embodiment project, producing a structured skill space. At test time, adapting to a new task requires only a single expert demonstration. The corresponding skill representation is then inferred via a lightweight convex optimization problem that minimizes the L1 action error, without requiring gradient updates. This gradient-free adaptation incurs minimal overhead while enabling rapid instantiation of new skills. Empirically, MoS-VLA achieves lower action-prediction error on five out of five unseen datasets and succeeds in both simulation and real-robot tasks where a pretrained VLA model fails outright. Project page: mos-vla.github.io/

Figures (7)

• Figure 1: In-context adaptation with function encoders. (Left) A human expert collects one trajectory of data in the new context. (Middle) The model is calibrated by solving a small, least absolute error optimization problem. This process takes only a few seconds, even on an RTX 3090. (Right) The adapted model autonomously executes actions in the new environment.
• Figure 2: Discrete vs. Continuous Skill Primitives.(Left) A discrete policy restricted to “right” or “up” yields inefficient, Manhattan-style paths and cannot reach all goals. (Right) Continuous combinations of skills enable direct movement in any direction, allowing the robot to reach all goals.
• Figure 3: The MoS-VLA Architecture and Training Pipeline. (Left) MoS-VLA builds on the OpenVLA backbone where images and instructions are processed by a Llama 2 model. The language modeling head is replaced with $k$ basis action heads. The action predictions are linearly combined via vector operations as described in Section \ref{['section:background_fe']}. (Right) During training, we iterate between context estimation and parameter updates. We utilize parallelism in both stages: during calibration, each training node computes and broadcasts the coefficient only for the datasets it hosts. During model updates, the context coefficients are held constant and trainable weights are adjusted with DDP.
• Figure 4: Validation Action $\mathbf{L^1}$ Error Across Train and OOD Datasets. We train MoS-VLA on the same Open X-Embodiment Magic Soup Plus data mixture as OpenVLA. Each subset is annotated with its proportion in the training set. Datasets marked as OOD are unseen during training. MoS-VLA, calibrated on only 512 training examples, achieves lower error than OpenVLA on 18 of 27 training subsets and on all 5 OOD datasets.
• Figure 5: Illustration of simulated and real robot tasks. MoS-VLA rollouts across five tasks in both simulated and on-robot experiments. Having never seen these tasks during training, MoS-VLA needs only one demonstration for gradient-free adaptation. Note that the objective for goal reaching is to hover above the block.