Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model

TL;DR

The Anchor Former (AcFormer), a novel vision-language connector designed to leverage the rich prior knowledge obtained from these visual anchors during pretraining, guiding the aggregation of information is introduced.

Abstract

In the realm of Multimodal Large Language Models (MLLMs), vision-language connector plays a crucial role to link the pre-trained vision encoders with Large Language Models (LLMs). Despite its importance, the vision-language connector has been relatively less explored. In this study, we aim to propose a strong vision-language connector that enables MLLMs to achieve high accuracy while maintain low computation cost. We first reveal the existence of the visual anchors in Vision Transformer and propose a cost-effective search algorithm to extract them. Building on these findings, we introduce the Anchor Former (AcFormer), a novel vision-language connector designed to leverage the rich prior knowledge obtained from these visual anchors during pretraining, guiding the aggregation of information. Through extensive experimentation, we demonstrate that the proposed method significantly reduces computational costs by nearly two-thirds compared with baseline, while simultaneously outperforming baseline methods. This highlights the effectiveness and efficiency of AcFormer. Codes are available at https://github.com/liuhaogeng/Anchor-Former.

Figures (7)

• Figure 1: Comparison of the average normalized accuracy (MMB, TextVQA, GQA). PR means Perceiver Resampler, which utilize the learnable query as information aggregator. Our method achieves highest accuracy comparing with the others while maintaining high training speed.
• Figure 2: Visualizations of the visual feature map and attention map pertaining to the [CLS] token. Here we select 10 layers in Vision Transformer to show their output. We present the attention maps corresponding to the [CLS] token in the final layer. Notably, special tokens within both the feature map and attention map are identified using red circles. These marked points are referred to as "visual anchors". Details can be found in Section \ref{['visual_anchor']}.
• Figure 3: Visualization of Anchor Former (AcFormer). We propose our token selection algorithm code in detail at Section \ref{['detailed_alg']}.
• Figure 4: Visualization of the Openai CLIP. Figure \ref{['visual1']} shows the visualization of EVA CLIP. We show that the phenomenon happens in different CLIP.
• Figure 5: Visualization of the attention map from the pre-trained Flamingo model (Removing the Perceiver Resampler). The attention map is the generated text's attending to the corresponding image patches.