Vision-Centric Activation and Coordination for Multimodal Large Language Models

TL;DR

This work targets the underutilization of vision-centric information in multimodal LLMs (MLLMs) by introducing VaCo, a framework that activates and coordinates visual priors from multiple vision foundation models. VaCo employs learnable Modular Task Queries (MTQs) and Visual Alignment Layers (VALs) to generate task-specific visual cues (VAQs) and uses a Token Gateway Mask (TGM) to prevent cross-task conflicts during causal decoding. The training objective combines standard MLLM loss with discriminative perceptual losses across several VFMs, enabling visual priors to guide text generation without multiple encoders at inference. Across general, region-level, and scene-level benchmarks, VaCo demonstrates superior visual-language understanding with reduced computational overhead compared to multi-encoder baselines, underscoring the value of vision-centric activation and coordination for MLLMs.

Abstract

Multimodal large language models (MLLMs) integrate image features from visual encoders with LLMs, demonstrating advanced comprehension capabilities. However, mainstream MLLMs are solely supervised by the next-token prediction of textual tokens, neglecting critical vision-centric information essential for analytical abilities. To track this dilemma, we introduce VaCo, which optimizes MLLM representations through Vision-Centric activation and Coordination from multiple vision foundation models (VFMs). VaCo introduces visual discriminative alignment to integrate task-aware perceptual features extracted from VFMs, thereby unifying the optimization of both textual and visual outputs in MLLMs. Specifically, we incorporate the learnable Modular Task Queries (MTQs) and Visual Alignment Layers (VALs) into MLLMs, activating specific visual signals under the supervision of diverse VFMs. To coordinate representation conflicts across VFMs, the crafted Token Gateway Mask (TGM) restricts the information flow among multiple groups of MTQs. Extensive experiments demonstrate that VaCo significantly improves the performance of different MLLMs on various benchmarks, showcasing its superior capabilities in visual comprehension.

Figures (10)

• Figure 1: Visualization of Visual Activation Methods: (a) Attention Visualization. Utilizing samples from MMVP tong2024eyes, we investigate which visual regions dominate the final predictions of MLLMs. Specifically, since MLLMs are anticipated to deliver answers at the final position of the input sequence, we visualize the attention scores between the last token and all image tokens. Compared to the other schemes (i.e., (i) LLaVA liu2023visual, (iii) multi-encoder scheme tong2024cambrian, (iv) reconstructive scheme wang2024reconstructive and (iv) the proposed VaCo), our vision-centric activation effectively directs MLLMs to focus on the crucial visual areas mentioned in the questions (Q) (e.g., "palm" and "fingers" in the first example), leading to correct answers; (b) Inference Example. VaCo facilitates MLLM in delivering comprehensive answers (A) through the assistance of diverse Visual Results. Note that visual outputs are an optional byproduct.
• Figure 2: Textual Loss Visualization across two stages. We investigate the impact of visual signal activation on text output: (a) At Pre-Training (PT) stage, the integration of a pure vision model in VaCo complicates initial alignment, but the loss convergence is similar to others; and (b) at Supervised Fine-Tuning (SFT) stage, the textual loss of VaCo converges more rapidly to a lower value, highlighting that query-based visual activation and coordination effectively facilitate MLLM acquisition of comprehensive visual information.
• Figure 3: Framework Overview of our VaCo. (a) We define the Modular Task Queries (MTQs) for various vision tasks, which jointly input the LLM with image and text tokens. The Visual Alignment Layer (VAL) directs these queries towards specific VFMs (e.g., Depth Anything yang2024depth2). We develop Token Gateway Mask (TGM) for causal attention to prevent feature conflicts across multiple MTQs. (b) Detailed architecture of VAL, with each equipped with an independent Visual Alignment Queries (VQAs), establishes a collaborative mechanism with MTQs.
• Figure 4: Illustration of the Token Gateway Mask (TGM). Here we present the TGM varying the number of visual tasks, i.e., $N=1,2,3$. Above illustrates the construction of TGM, while below depicts the information flow within a causal transformer utilizing TGM. We represent the image token in blue, while different types of MTQs are depicted in orange, yellow and pink, respectively. Note that we omit the text tokens for simplicity.
• Figure 5: Causal Perception Results of our VaCo. For each tuple corresponding to the input original image, we present the output from the MLLM task queries and the output from the pure visual foundation model (right), including DINOv2 oquab2023dinov2, DepthAnything V2 yang2024depth2, OneFormer jain2023oneformer, and VGGT wang2025vggt. The aligned perception results suggest that VaCo effectively pre-summarizes specific visual priors via task queries in visual understanding, thereby enhancing the correctness of text output through the causal mechanism.