AMoE: Agglomerative Mixture-of-Experts Vision Foundation Model

TL;DR

AMoE tackles the data inefficiency of multi-teacher distillation for vision foundation models by distilling SigLIP2 and DINOv3 into a Mixture-of-Experts student. It introduces token-balanced batching, asymmetric relational knowledge distillation, and hierarchical OpenLVD200M data curation to stabilize learning and improve sample efficiency, achieving strong global representations and competitive dense-feature performance with far less data than prior MT models. The approach is validated through extensive experiments, ARKD and CKA-based analyses, and a two-stage high-resolution training regime, demonstrating scalable, data-efficient vision foundation modeling. The work also shows that specialized MoE routing across teachers yields emergent expert specialization and practical benefits for grounding VLMs with limited annotations. Overall, AMoE provides a practical pathway to high-quality, scalable vision foundation models with reduced data and compute requirements.

Abstract

Vision foundation models trained via multi-teacher distillation offer a promising path toward unified visual representations, yet the learning dynamics and data efficiency of such approaches remain underexplored. In this paper, we systematically study multi-teacher distillation for vision foundation models and identify key factors that enable training at lower computational cost. We introduce Agglomerative Mixture-of-Experts Vision Foundation Models (AMoE), which distill knowledge from SigLIP2 and DINOv3 simultaneously into a Mixture-of-Experts student. We show that (1) our Asymmetric Relation-Knowledge Distillation loss preserves the geometric properties of each teacher while enabling effective knowledge transfer, (2) token-balanced batching that packs varying-resolution images into sequences with uniform token budgets stabilizes representation learning across resolutions without sacrificing performance, and (3) hierarchical clustering and sampling of training data--typically reserved for self-supervised learning--substantially improves sample efficiency over random sampling for multi-teacher distillation. By combining these findings, we curate OpenLVD200M, a 200M-image corpus that demonstrates superior efficiency for multi-teacher distillation. Instantiated in a Mixture-of-Experts. We release OpenLVD200M and distilled models.

Figures (8)

• Figure 1: AMoE vision foundation model: A Mixture-of-Experts student is distilled from multiple frozen vision teachers as shown in the multi-teacher distillation stage (on the left). The input image is fed to both teachers (SigLIP2 and DINOv3) and the student to obtain respective patch and global representation embeddings. Additional register tokens are employed in the student model, similar to DINOv3. The student embeddings are then projected to individual teacher embedding spaces via learnable teacher-specific heads. The learning objective includes matching the patch and global (CLS) embeddings of the student with corresponding embeddings of both teachers, in addition to matching the register embeddings with DINOv3 teacher. Moreover, we introduce an asymmetric relational knowledge distillation loss for matching pairwise geometry among samples. The PCA map of the student embeddings (at the top) illustrates the high-quality, dense representations obtained after distillation.
• Figure 2: Token-balanced batching: Packing multiple native-resolution images per sequence up to a fixed token budget and applying FlexAttention masks to prevent inter-image attention stabilizes multi-resolution training, prevents low-res forgetting, and improves performance. This strategy also allows for more resource-efficient training with less padding; we go from 7.5k to 20k tokens per second.
• Figure 3: Linear CKA alignments between MoE experts and teacher layers at several AMoE layers.
• Figure 4: We visualize PCA projections of global features, patches, and DINOv3 registers (0 and 1): original data (Col 1), synthetic Gaussian data generated from estimated moments (Col 2), and their respective versions after Phi-S transformation (Cols 3 and 4). While global, patch embeddings, and the 0th register are well-approximated by Gaussian statistics and effectively whitened by Phi-S, the first register exhibits multi-mode distributions (Row 4) where simple moments capture inter-mode statistics. Hence, applying Phi-S to this register yields incorrect transformations.
• Figure 5: Impact of Asymmetric Relational Knowledge Distillation (ARKD) on training dynamics.