Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

3D-Mix for VLA: A Plug-and-Play Module for Integrating VGGT-based 3D Information into Vision-Language-Action Models

Bin Yu, Shijie Lian, Xiaopeng Lin, Zhaolong Shen, Yuliang Wei, Haishan Liu, Changti Wu, Hang Yuan, Bailing Wang, Cong Huang, Kai Chen

Abstract

Vision-Language-Action (VLA) models leverage Multimodal Large Language Models (MLLMs) for robotic control, but recent studies reveal that MLLMs exhibit limited spatial intelligence due to training predominantly on 2D data, resulting in inadequate 3D perception for manipulation tasks. While recent approaches incorporate specialized 3D vision models such as VGGT to enhance spatial understanding, they employ diverse integration mechanisms without systematic investigation, leaving the optimal fusion strategy unclear. We conduct a comprehensive pilot study comparing nine VGGT integration schemes on standardized benchmarks and find that semantic-conditioned gated fusion, which adaptively balances 2D semantic and 3D geometric features based on task context, achieved the strongest performance among all nine evaluated fusion schemes in our pilot study. We present 3D-Mix, a plug-and-play module that integrates into diverse VLA architectures (GR00T-style and $π$-style) without modifying existing MLLM or action expert components. Experiments across six MLLM series (nine model variants, 2B--8B parameters) on SIMPLER and LIBERO show that 3D-Mix delivers consistent performance gains, averaging +7.0% on the out-of-domain (OOD) SIMPLER benchmark across all nine GR00T-style variants, establishing a principled approach for enhancing spatial intelligence in VLA systems.

3D-Mix for VLA: A Plug-and-Play Module for Integrating VGGT-based 3D Information into Vision-Language-Action Models

Abstract

Vision-Language-Action (VLA) models leverage Multimodal Large Language Models (MLLMs) for robotic control, but recent studies reveal that MLLMs exhibit limited spatial intelligence due to training predominantly on 2D data, resulting in inadequate 3D perception for manipulation tasks. While recent approaches incorporate specialized 3D vision models such as VGGT to enhance spatial understanding, they employ diverse integration mechanisms without systematic investigation, leaving the optimal fusion strategy unclear. We conduct a comprehensive pilot study comparing nine VGGT integration schemes on standardized benchmarks and find that semantic-conditioned gated fusion, which adaptively balances 2D semantic and 3D geometric features based on task context, achieved the strongest performance among all nine evaluated fusion schemes in our pilot study. We present 3D-Mix, a plug-and-play module that integrates into diverse VLA architectures (GR00T-style and -style) without modifying existing MLLM or action expert components. Experiments across six MLLM series (nine model variants, 2B--8B parameters) on SIMPLER and LIBERO show that 3D-Mix delivers consistent performance gains, averaging +7.0% on the out-of-domain (OOD) SIMPLER benchmark across all nine GR00T-style variants, establishing a principled approach for enhancing spatial intelligence in VLA systems.

Paper Structure

This paper contains 23 sections, 13 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language-Action Models.
  4. 3D Fusion for VLA Models.
  5. Pilot Study: Comparative Analysis of VGGT Fusion Schemes
  6. Base VLA Architecture
  7. Multiple VGGT Fusion Schemes
  8. Experimental Setup for Comparative Evaluation
  9. Pilot Study Results and Analysis
  10. 3D-Mix Module
  11. Module Design
  12. Integration with GR00T-style Architecture
  13. Integration with $\pi$-style Architecture
  14. Implementation as a Plug-and-Play Module
  15. Experiment
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Overview of nine VGGT fusion schemes evaluated in our pilot study. Each scheme integrates VGGT-derived 3D geometric features into the VLA pipeline at a different location or via a different mechanism, ranging from early token-level injection to training-time alignment and cross-attention fusion.
  • Figure 2: Integration of 3D-Mix into GR00T-style and $\pi$-style VLA Architectures
  • Figure 3: Ablation study results. Each subfigure corresponds to one ablation experiment: (a) VGGT Frozen vs. Trainable. (b) 3D Information Sensitivity. (c) Sparse Layer Fusion Depth ($\pi$-style).