Expertise need not monopolize: Action-Specialized Mixture of Experts for Vision-Language-Action Learning

TL;DR

AdaMoE addresses the challenge of scaling Vision-Language-Action models with limited robot data by decoupling expert selection from weighting through a scale adapter, enabling collaborative use of action experts while inheriting pretrained VLA weights. The method introduces Shared and Routed experts, a top-$k$ router, and a scale-adapter that adds to router outputs, preserving load balancing while improving task-specific contributions. Empirical results show consistent gains on LIBERO (1.8%), RoboTwin 2.0 (9.3%), and real-world tasks (average 21.5%), with notable improvements in long-horizon and domain-randomized settings. The work demonstrates a practical pathway to scale pretrained VLA models for robust robotic manipulation, balancing performance and inference efficiency.

Abstract

Vision-Language-Action (VLA) models are experiencing rapid development and demonstrating promising capabilities in robotic manipulation tasks. However, scaling up VLA models presents several critical challenges: (1) Training new VLA models from scratch demands substantial computational resources and extensive datasets. Given the current scarcity of robot data, it becomes particularly valuable to fully leverage well-pretrained VLA model weights during the scaling process. (2) Real-time control requires carefully balancing model capacity with computational efficiency. To address these challenges, We propose AdaMoE, a Mixture-of-Experts (MoE) architecture that inherits pretrained weights from dense VLA models, and scales up the action expert by substituting the feedforward layers into sparsely activated MoE layers. AdaMoE employs a decoupling technique that decouples expert selection from expert weighting through an independent scale adapter working alongside the traditional router. This enables experts to be selected based on task relevance while contributing with independently controlled weights, allowing collaborative expert utilization rather than winner-takes-all dynamics. Our approach demonstrates that expertise need not monopolize. Instead, through collaborative expert utilization, we can achieve superior performance while maintaining computational efficiency. AdaMoE consistently outperforms the baseline model across key benchmarks, delivering performance gains of 1.8% on LIBERO and 9.3% on RoboTwin. Most importantly, a substantial 21.5% improvement in real-world experiments validates its practical effectiveness for robotic manipulation tasks.

Figures (7)

• Figure 1: AdaMoE architecture overview. (a) Overall Pipeline: Multi-modal input processing through VLM backbone and transformer blocks with integrated MoE layers. (b) Expert Initialization: Shared expert inherits original FFN weights while routed experts are created as copies for efficient scaling. (c) Vanilla MoE: Single router couples expert selection and weighting through top-k selection and softmax outputs. (d) AdaMoE (Ours): Decoupled architecture with independent router (blue) for selection and scale adapter (green) for additional weighting, including shared experts (SE) and routed experts for flexible utilization.
• Figure 2: Visualization of expert usage intensity
• Figure 3: Architecture variants for decoupling expert selection and weighting. (a) Vanilla MoE couples selection and weighting through a single router. (b) CSMoE concatenates router outputs with action tokens for scale adaptation. (c) AdaMoE (Ours) additively combines independent router and scale adapter weights, achieving decoupling of expert selection from contribution weighting.
• Figure 4: Hardware
• Figure 5: Manipulation task demonstrations: bowl stacking, bottle adjustment, bell pressing, and cup placement.