MoCHA: Advanced Vision-Language Reasoning with MoE Connector and Hierarchical Group Attention

TL;DR

MoCHA tackles the challenge of efficient vision-language reasoning by integrating four heterogeneous vision encoders with a sparse Mixture of Experts Connector and Hierarchical Group Attention. The MoECs enable dynamic, dimension-aware routing of visual features to specialized experts, while HGA fuses multi-encoder representations with adaptive gating. Through a two-stage training regimen on Phi2-2.7B and Vicuna-7B, MoCHA achieves strong open-weight VLLM performance, reduces visual hallucinations on POPE, and demonstrates favorable efficiency (lower GFLOPs and fast inference) compared to larger baselines. The results validate dynamic, multi-encoder fusion as a practical and effective strategy for vision-centric LLMs in real-world multimodal tasks.

Abstract

Vision large language models (VLLMs) are focusing primarily on handling complex and fine-grained visual information by incorporating advanced vision encoders and scaling up visual models. However, these approaches face high training and inference costs, as well as challenges in extracting visual details, effectively bridging across modalities. In this work, we propose a novel visual framework, MoCHA, to address these issues. Our framework integrates four vision backbones (i.e., CLIP, SigLIP, DINOv2 and ConvNeXt) to extract complementary visual features and is equipped with a sparse Mixture of Experts Connectors (MoECs) module to dynamically select experts tailored to different visual dimensions. To mitigate redundant or insufficient use of the visual information encoded by the MoECs module, we further design a Hierarchical Group Attention (HGA) with intra- and inter-group operations and an adaptive gating strategy for encoded visual features. We train MoCHA on two mainstream LLMs (e.g., Phi2-2.7B and Vicuna-7B) and evaluate their performance across various benchmarks. Notably, MoCHA outperforms state-of-the-art open-weight models on various tasks. For example, compared to CuMo (Mistral-7B), our MoCHA (Phi2-2.7B) presents outstanding abilities to mitigate hallucination by showing improvements of 3.25% in POPE and to follow visual instructions by raising 153 points on MME. Finally, ablation studies further confirm the effectiveness and robustness of the proposed MoECs and HGA in improving the overall performance of MoCHA.

Figures (6)

• Figure 1: Architecture of MoCHA. MoCHA integrates multiple vision encoders (i.e., CLIP, SigLIP, ConvNeXt, and DINOv2), and consists of a Mixture of Experts Connectors (MoECs) module and Hierarchical Group Attention (HGA) module.
• Figure 2: Sparse Top-$K$ MoEC block. Replacing MLP connector with MoEC for visual-text token alignment.
• Figure 3: Intra-group Attention. Each model selects and aggregates the Top-$M$ features for its tokens within the same group.
• Figure 4: Inter-group Attention. SigLIP selects and aggregates the Top-$N$ features for each token across different models (groups).
• Figure 5: Ablation of Channel vs. Sequential Concatenation for MoECs with four encoders.