HMVLA: Hyperbolic Multimodal Fusion for Vision-Language-Action Models

TL;DR

This work addresses semantic misalignment in Vision-Language-Action models caused by fine-tuning pre-trained VLMs, which can disrupt hierarchical image-text semantics during action grounding. It proposes HMVLA, a hyperbolic multimodal fusion framework that embeds features in the Lorentz hyperboloid to capture hierarchical structure and employs a sparsely gated Mixture-of-Experts for adaptive cross-modal routing. Key contributions include a hyperbolic semantic alignment with an entailment-cone constraint and a balanced MoE objective, validated on LIBERO and a reconstructed Gen dataset to demonstrate robust generalization. Results show improved task accuracy and cross-domain adaptability, highlighting hyperbolic geometry as a viable foundation for robust VLA systems in robotics.

Abstract

Vision Language Action (VLA) models have recently shown great potential in bridging multimodal perception with robotic control. However, existing methods often rely on direct fine-tuning of pre-trained Vision-Language Models (VLMs), feeding semantic and visual features directly into a policy network without fully addressing the unique semantic alignment challenges in the VLA domain. In this paper, we propose HMVLA, a novel VLA framework that exploits the inherent hierarchical structures in vision and language for comprehensive semantic alignment. Unlike traditional methods that perform alignment in Euclidean space, our HMVLA embeds multimodal features in hyperbolic space, enabling more effective modeling of the hierarchical relationships present in image text data. Furthermore, we introduce a sparsely gated Mixture of Experts (MoE) mechanism tailored for semantic alignment, which enhances multimodal comprehension between images and text while improving efficiency. Extensive experiments demonstrate that HMVLA surpasses baseline methods in both accuracy and generalization. In addition, we validate its robustness by reconstructing datasets to further test cross domain adaptability.

Figures (5)

• Figure 1: Comparison of hyperbolic representation (Left) and Euclidean contrastive Loss (Right) for VLA Alignment.
• Figure 2: Overview of our HMVLA framework. A hierarchical VLA transformer with hyperbolic projection and a sparsely-gated Mixture-of-Experts (MoE) for enhanced semantic alignment.
• Figure 3: Comparison of HMVLA and Baseline Models on Spatial, Object, Goal, and Our Constructed Datasets.
• Figure 4: The HMVLA model's grasping process on LIBERO benchmark.
• Figure 5: Ablation Experiments Evaluating Hyperbolic, MoE, and Cross-Attention.