MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agent

TL;DR

This work addresses the challenge of extending Vision-Language-Action models to multi-skill generalists by identifying two primary sources of non-mergeability: destructive LoRA parameter interference in the VLM backbone and cross-task dependencies in the action expert. The authors propose MergeVLA, a merge-friendly VLA architecture that uses sparse, task-specific LoRA masking and a cross-attention-only action head, along with a test-time, unsupervised task router to handle unknown tasks. Across LIBERO, LIBERO-Plus, RoboTwin, and real-world SO101 experiments, MergeVLA achieves multi-task performance comparable to or exceeding individually finetuned experts, with strong robustness to distribution shifts and cross-embodiment generalization. The results demonstrate that principled architectural redesigns and lightweight routing can enable scalable merging for embodied generalists, offering a practical path toward generalist VLAs. Future work will explore larger backbones and diverse pretraining to further improve merging effectiveness.

Abstract

Recent Vision-Language-Action (VLA) models reformulate vision-language models by tuning them with millions of robotic demonstrations. While they perform well when fine-tuned for a single embodiment or task family, extending them to multi-skill settings remains challenging: directly merging VLA experts trained on different tasks results in near-zero success rates. This raises a fundamental question: what prevents VLAs from mastering multiple skills within one model? With an empirical decomposition of learnable parameters during VLA fine-tuning, we identify two key sources of non-mergeability: (1) Finetuning drives LoRA adapters in the VLM backbone toward divergent, task-specific directions beyond the capacity of existing merging methods to unify. (2) Action experts develop inter-block dependencies through self-attention feedback, causing task information to spread across layers and preventing modular recombination. To address these challenges, we present MergeVLA, a merging-oriented VLA architecture that preserves mergeability by design. MergeVLA introduces sparsely activated LoRA adapters via task masks to retain consistent parameters and reduce irreconcilable conflicts in the VLM. Its action expert replaces self-attention with cross-attention-only blocks to keep specialization localized and composable. When the task is unknown, it uses a test-time task router to adaptively select the appropriate task mask and expert head from the initial observation, enabling unsupervised task inference. Across LIBERO, LIBERO-Plus, RoboTwin, and multi-task experiments on the real SO101 robotic arm, MergeVLA achieves performance comparable to or even exceeding individually finetuned experts, demonstrating robust generalization across tasks, embodiments, and environments.

Figures (9)

• Figure 1: Comparison between the structures of different VLAs. OpenVLA uses a standard VLM for token-based action generation. VLA-Adapter adds an action expert with cross- and self-attention layers. MergeVLA simplifies this design by removing non-mergeable self-attention layers for effective merging.
• Figure 2: Overview of MergeVLA architecture. (1) To address destructive LoRA parameter interference in finetuned VLM, task masks are applied to all merged LoRA modules to selectively activate the merged parameters contributing to task-relevant responses while suppressing those that mislead other tasks. (2) To solve the incompatibility of action experts, the architecture is redesigned to contain only cross-attention blocks and use $\mathrm{sigmoid}$ gate to preserve and rely on robust VLM features. Most blocks then can be merged except deeper blocks named expert head are left unmerged due to their task specification. (3) To address the setting where the task identity is unknown at inference time, a training-free test-time task router is adopted to dynamically select task-specified components by computing task relevance from VLM hidden states in the value-based subspace of the merged action expert.
• Figure 3: Left: Selfish ratio of the masks from TA TA and TIES ties by merging different numbers of tasks. The selfish ratio is computed following Equation \ref{['eq:selfish ratio']}. Right: The average relative L2 distance across blocks between all pairs of action experts.
• Figure 4: Seven perturbation types in the LIBERO-Plus benchmark, used to evaluate robustness under visual and language shifts.
• Figure 5: Experimental setup in the RoboTwin environment, featuring three robotic embodiments and a suite of manipulation tasks for cross-embodiment evaluation.