Astrea: A MOE-based Visual Understanding Model with Progressive Alignment

TL;DR

Astrea tackles the challenge of balancing heterogeneous experts in MoE-based Vision-Language Models for broad visual understanding. It introduces progressive pre-alignment to initialize four task-specific experts (detection, segmentation, classification, captioning) and a dynamic fusion mechanism guided by a router and adapters, complemented by momentum-contrast learning. Across 12 benchmarks, Astrea delivers a +4.7% average improvement over state-of-the-art methods, illustrating effective cross-task knowledge sharing while preserving task specialization. The approach offers a scalable path toward general-purpose multimodal agents and provides methodological insights for coordinating diverse visual experts in large-scale VLMs.

Abstract

Vision-Language Models (VLMs) based on Mixture-of-Experts (MoE) architectures have emerged as a pivotal paradigm in multimodal understanding, offering a powerful framework for integrating visual and linguistic information. However, the increasing complexity and diversity of tasks present significant challenges in coordinating load balancing across heterogeneous visual experts, where optimizing one specialist's performance often compromises others' capabilities. To address task heterogeneity and expert load imbalance, we propose Astrea, a novel multi-expert collaborative VLM architecture based on progressive pre-alignment. Astrea introduces three key innovations: 1) A heterogeneous expert coordination mechanism that integrates four specialized models (detection, segmentation, classification, captioning) into a comprehensive expert matrix covering essential visual comprehension elements; 2) A dynamic knowledge fusion strategy featuring progressive pre-alignment to harmonize experts within the VLM latent space through contrastive learning, complemented by probabilistically activated stochastic residual connections to preserve knowledge continuity; 3) An enhanced optimization framework utilizing momentum contrastive learning for long-range dependency modeling and adaptive weight allocators for real-time expert contribution calibration. Extensive evaluations across 12 benchmark tasks spanning VQA, image captioning, and cross-modal retrieval demonstrate Astrea's superiority over state-of-the-art models, achieving an average performance gain of +4.7\%. This study provides the first empirical demonstration that progressive pre-alignment strategies enable VLMs to overcome task heterogeneity limitations, establishing new methodological foundations for developing general-purpose multimodal agents.

Figures (6)

• Figure 1: The pipeline of the Astrea. The left panel illustrates the four stages of pre-alignment, with the Instruct phase not explicitly shown. Note that the target labels simply highlight the emphasized points; in reality, the target scope is gradually expanded. The right panel presents the main model, which is initialized with the outcomes from the left panel.
• Figure 2: The demo of the QA task. The bar chart shows the contribution probabilities of the four models output by the router.
• Figure 3: The trend of various metrics for different subjects across training steps during the alignment stage. The $MME^P$ results are normalized to a scale of 0-100. The alignment proceeds sequentially through the Caption, Classification, Detection, and Segmentation stages.
• Figure 4: Demo of Image Description (left), Landmark Recognition (right) and General Knowledge (bottom).
• Figure 5: Demo of Chart Understanding.