AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning

TL;DR

This work tackles the memory and applicability barriers of multimodal in-context learning (ICL) for large language models by introducing AIM, a lightweight framework that aggregates image information from multimodal demonstrations into the dense textual latent space. By freezing the backbone MLLM and training a small projection layer, AIM converts multimodal demonstrations into text-like prompts that guide generation, enabling efficient ICL across backbones. Empirical results on Flickr30k, OKVQA, VizWiz, and Hateful Memes show AIM achieves comparable or superior performance with dramatically fewer tokens and improved throughput and memory efficiency, highlighting practical applicability for real-world multimodal tasks. The approach also supports demonstration caching via a demonstration bank and demonstrates robustness across datasets, indicating significant practical impact for deploying multimodal ICL in resource-constrained settings.

Abstract

In-context learning (ICL) facilitates Large Language Models (LLMs) exhibiting emergent ability on downstream tasks without updating billions of parameters. However, in the area of multi-modal Large Language Models (MLLMs), two problems hinder the application of multi-modal ICL: (1) Most primary MLLMs are only trained on single-image datasets, making them unable to read multi-modal demonstrations. (2) With the demonstrations increasing, thousands of visual tokens highly challenge hardware and degrade ICL performance. During preliminary explorations, we discovered that the inner LLM tends to focus more on the linguistic modality within multi-modal demonstrations to generate responses. Therefore, we propose a general and light-weighted framework \textbf{AIM} to tackle the mentioned problems through \textbf{A}ggregating \textbf{I}mage information of \textbf{M}ultimodal demonstrations to the dense latent space of the corresponding linguistic part. Specifically, AIM first uses the frozen backbone MLLM to read each image-text demonstration and extracts the vector representations on top of the text. These vectors naturally fuse the information of the image-text pair, and AIM transforms them into fused virtual tokens acceptable for the inner LLM via a trainable projection layer. Ultimately, these fused tokens function as variants of multi-modal demonstrations, fed into the MLLM to direct its response to the current query as usual. Because these fused tokens stem from the textual component of the image-text pair, a multi-modal demonstration is nearly reduced to a pure textual demonstration, thus seamlessly applying to any MLLMs. With its de facto MLLM frozen, AIM is parameter-efficient and we train it on public multi-modal web corpora which have nothing to do with downstream test tasks.

Figures (11)

• Figure 1: Memory cost comparison between AIM and LLaVA-Next on Flickr30k. The memory cost of LLaVA-Next occurs a surge, while it almost remains unchanged in AIM.
• Figure 2: Performance comparison between AIM and its underlying backbone in the 16-shot ICL setting.
• Figure 3: The hot map of attention scores when QWen-VL generates the first token on the hateful memes dataset. The brighter represents that responses to be generated have paid more attention to the current visual/textual tokens. Obviously, the generation relies more on the textual part of a multimodal demonstration
• Figure 4: The architecture of AIM. Fused tokens from different demonstrations are concatenated and fed into the inner LLM, discarding original visual tokens.
• Figure 5: The perplexity variation tendency corresponds to the number of demonstrations. 0-shot server as the baseline.