Scope: Selective Cross-modal Orchestration of Visual Perception Experts

TL;DR

Scope introduces a dynamic Mixture-of-Encoders for vision-language models, where a lightweight router selects a single encoder from a pool of $K$ routed encoders based on instance-level cues from a shared visual feature map $V_s$ and an encoded text prompt $T$. The selected encoder’s features are fused with the shared encoder’s output to form a visual prefix that conditions the LLM, while a dual-entropy and dual-auxiliary loss framework trains the router to balance dataset-wide encoder usage with per-instance confidence. Empirically, Scope-CA with one shared plus one routed encoder achieves higher accuracy than configurations that activate all encoders, while reducing compute by 24–49% across eight benchmarks, including VCR and document-visual QA tasks. The approach demonstrates that intelligent, context-aware encoder selection can outperform brute-force multi-encoder fusion, offering a scalable, efficient paradigm for future multimodal systems.

Abstract

Vision-language models (VLMs) benefit from multiple vision encoders, but naively stacking them yields diminishing returns while multiplying inference costs. We propose SCOPE, a Mixture-of-Encoders (MoEnc) framework that dynamically selects one specialized encoder per image-text pair via instance-level routing, unlike token-level routing in traditional MoE. SCOPE maintains a shared encoder and a pool of routed encoders. A lightweight router uses cross-attention between text prompts and shared visual features to select the optimal encoder from the routed encoders. To train this router, we introduce dual entropy regularization with auxiliary losses to balance dataset-level load distribution with instance-level routing confidence. Remarkably, SCOPE with one shared plus one routed encoder outperforms models using all four extra encoders simultaneously, while reducing compute by 24-49\%. This demonstrates that intelligent encoder selection beats brute-force aggregation, challenging the prevailing paradigm in multi-encoder VLMs.

Figures (2)

• Figure 1: Our model, Scope (marked by $\star$), is compared against baseline VLMs configured with zero to four fixed extra vision encoders. For each generation, Scope is architecturally equivalent to a single–extra-encoder model but uses a router to dynamically choose that encoder. This dynamic approach allows Scope to achieve superior performance across all tasks, notably surpassing the memory-intensive four-encoder model, especially on the VCR_EN_HARD dataset. Please refer to Table \ref{['tab:model_performance']} for detailed benchmark scores.
• Figure 2: Overview of the dynamic VLM architecture in Scope. An input image and user query are processed to generate answers. The architecture features a Shared Image Encoder and a Language Embedding module that generate a Shared Image Embedding and a Language Embedding, respectively. These embeddings are fed into a Router that dynamically selects one Routed Image Encoder from a pool of $K$ expert encoders. The selected auxiliary encoder then generates a Routed Image Embedding. This and Shared Image Embedding is merged into Merged Image Embedding. Finally, this merged embedding is input into a Large Language Model (LLM) to produce the Output Answers. The diagram also highlights that certain modules are Pretrained Modules (cyan), others are Initialized Modules (purple), and the data flowing through the system are Tensors (pink). The dashed lines indicate the selection process, where only one path is chosen. On the right, the microscope serves as a visual metaphor for the model's function. It illustrates how Scope "zooms in" on a task by routing the input to a selected encoder, akin to a researcher using a microscope to examine a sample with an appropriate objective lens.