Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TwinBrainVLA: Unleashing the Potential of Generalist VLMs for Embodied Tasks via Asymmetric Mixture-of-Transformers

Bin Yu, Shijie Lian, Xiaopeng Lin, Yuliang Wei, Zhaolong Shen, Changti Wu, Yuzhuo Miao, Xinming Wang, Bailing Wang, Cong Huang, Kai Chen

TL;DR

This work tackles catastrophic forgetting in Vision-Language-Action models by decoupling semantic understanding from embodied control. It introduces TwinBrainVLA, an asymmetric dual-VLM architecture with a frozen Left Brain for open-world reasoning and a trainable Right Brain for proprioception, fused through Asymmetric Mixture-of-Transformers and a Flow-Matching Action Expert. The Flow-Matching objective guides the Right Brain and the action predictor in leveraging rich Right Brain conditioning to generate smooth, continuous robot actions, while the Left Brain remains a semantic anchor. Across SimplerEnv and RoboCasa benchmarks, TwinBrainVLA achieves state-of-the-art or competitive manipulation performance while preserving the general semantic capabilities, demonstrating a practical path toward general-purpose robots with high-level understanding and low-level dexterity.

Abstract

Standard Vision-Language-Action (VLA) models typically fine-tune a monolithic Vision-Language Model (VLM) backbone explicitly for robotic control. However, this approach creates a critical tension between maintaining high-level general semantic understanding and learning low-level, fine-grained sensorimotor skills, often leading to "catastrophic forgetting" of the model's open-world capabilities. To resolve this conflict, we introduce TwinBrainVLA, a novel architecture that coordinates a generalist VLM retaining universal semantic understanding and a specialist VLM dedicated to embodied proprioception for joint robotic control. TwinBrainVLA synergizes a frozen "Left Brain", which retains robust general visual reasoning, with a trainable "Right Brain", specialized for embodied perception, via a novel Asymmetric Mixture-of-Transformers (AsyMoT) mechanism. This design allows the Right Brain to dynamically query semantic knowledge from the frozen Left Brain and fuse it with proprioceptive states, providing rich conditioning for a Flow-Matching Action Expert to generate precise continuous controls. Extensive experiments on SimplerEnv and RoboCasa benchmarks demonstrate that TwinBrainVLA achieves superior manipulation performance compared to state-of-the-art baselines while explicitly preserving the comprehensive visual understanding capabilities of the pre-trained VLM, offering a promising direction for building general-purpose robots that simultaneously achieve high-level semantic understanding and low-level physical dexterity.

TwinBrainVLA: Unleashing the Potential of Generalist VLMs for Embodied Tasks via Asymmetric Mixture-of-Transformers

TL;DR

This work tackles catastrophic forgetting in Vision-Language-Action models by decoupling semantic understanding from embodied control. It introduces TwinBrainVLA, an asymmetric dual-VLM architecture with a frozen Left Brain for open-world reasoning and a trainable Right Brain for proprioception, fused through Asymmetric Mixture-of-Transformers and a Flow-Matching Action Expert. The Flow-Matching objective guides the Right Brain and the action predictor in leveraging rich Right Brain conditioning to generate smooth, continuous robot actions, while the Left Brain remains a semantic anchor. Across SimplerEnv and RoboCasa benchmarks, TwinBrainVLA achieves state-of-the-art or competitive manipulation performance while preserving the general semantic capabilities, demonstrating a practical path toward general-purpose robots with high-level understanding and low-level dexterity.

Abstract

Standard Vision-Language-Action (VLA) models typically fine-tune a monolithic Vision-Language Model (VLM) backbone explicitly for robotic control. However, this approach creates a critical tension between maintaining high-level general semantic understanding and learning low-level, fine-grained sensorimotor skills, often leading to "catastrophic forgetting" of the model's open-world capabilities. To resolve this conflict, we introduce TwinBrainVLA, a novel architecture that coordinates a generalist VLM retaining universal semantic understanding and a specialist VLM dedicated to embodied proprioception for joint robotic control. TwinBrainVLA synergizes a frozen "Left Brain", which retains robust general visual reasoning, with a trainable "Right Brain", specialized for embodied perception, via a novel Asymmetric Mixture-of-Transformers (AsyMoT) mechanism. This design allows the Right Brain to dynamically query semantic knowledge from the frozen Left Brain and fuse it with proprioceptive states, providing rich conditioning for a Flow-Matching Action Expert to generate precise continuous controls. Extensive experiments on SimplerEnv and RoboCasa benchmarks demonstrate that TwinBrainVLA achieves superior manipulation performance compared to state-of-the-art baselines while explicitly preserving the comprehensive visual understanding capabilities of the pre-trained VLM, offering a promising direction for building general-purpose robots that simultaneously achieve high-level semantic understanding and low-level physical dexterity.
Paper Structure (13 sections, 7 equations, 2 figures, 2 tables)

This paper contains 13 sections, 7 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Motivation: Catastrophic Forgetting in VLA Models
  4. Method: TwinBrainVLA
  5. Asymmetric Dual-VLM as Backbone
  6. Flow-Matching Action Expert
  7. Training Strategy
  8. Experiment
  9. Experiments on SimplerEnv
  10. Experiments on RoboCasa
  11. Conclusion
  12. Training Hyperparameters
  13. VLM Prompt Template

Figures (2)

  • Figure 1: Architectural comparison between Vanilla VLA and TwinBrainVLA.
  • Figure 2: The framework of TwinBrainVLA.(a) Overall Architecture. The model features an Asymmetric Mixture-of-Transformers design composed of two distinct pathways: a frozen "Left Brain" (Generalist) for semantic reasoning and a trainable "Right Brain" (Specialist) for embodied motor control. The Right Brain fuses visual, textual, and proprioceptive state inputs to provide conditioning for the Action Expert, which utilizes a Flow-Matching algorithm to denoise continuous robotic actions. (b) Asymmetric MoT Mechanism (AsyMoT). Through causal self-attention, the trainable Right Brain attends to the frozen Key-Value (KV) pairs of the Left Brain, enabling the transfer of general semantic knowledge to the robotic control policy without catastrophic forgetting.