Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrations

TL;DR

<3-5 sentence high-level summary> Multimodal in-context learning in LVLMs is highly sensitive to how demonstrations are configured. The authors analyze the roles of Task Recognition (TR) and Task Learning (TL) and show that task semantics in the demonstrations guide robust cross-modal reasoning; they introduce SabER, a decoder-only transformer with a Task Guider that autoregressively selects and arranges ICDs from a demonstration library. SabER outperforms strong baselines across diverse VL tasks and LVLMs, with ablations highlighting the importance of a task-aware [TASK] prompt, hierarchical TG updates, and a sparsity-driven attention mask. The work provides a mechanism-driven approach to construct high-quality multimodal ICD sequences, improving robustness and generalization of multimodal ICL in real-world settings.

Abstract

Multimodal in-context learning (ICL) has emerged as a key capability of Large Vision-Language Models (LVLMs), driven by their increasing scale and applicability. Despite its promise, effective ICL in the multimodal setting remains challenging due to the inherent complexity of image-text inputs and the high sensitivity of ICL performance to input configurations. In this work, we shed light on the core mechanism underlying multimodal ICL, identifying task mapping as a crucial factor in configuring robust in-context demonstration (ICD) sequences. Building on these insights, we propose \textit{SabER}, a lightweight yet powerful decoder-only transformer equipped with task-aware attention, which intelligently selects and arranges ICDs from a demonstration library in an autoregressive fashion. This design enables fine-grained feature extraction and cross-modal reasoning, iteratively refining task mapping to generate high-quality ICD sequences. Through extensive experiments covering five LVLMs and nine benchmark datasets, SabER not only demonstrates strong empirical performance, but also provides deeper understanding of how task semantics interact with multimodal ICDs. Our findings highlight the importance of principled ICD sequence configuration and open new avenues to enhance multimodal ICL in a wide range of real-world scenarios.

Figures (4)

• Figure 1: Results of five settings across two tasks and two LVLMs which represent different parts of LVLM's in-context learning.
• Figure 2: The output of multimodal ICL evolves across layers in the LVLM given a 2-shot sequence (a). As illustrated by the pie charts in (b), processing a complete ICD sequence involves several distinct stages: capturing information from the query sample, identifying mappings within the ICD and engaging in in-depth reasoning, and ultimately leveraging the multimodal information to predict the output.
• Figure 3: Overview pipeline of our proposed model $SabER$.
• Figure 4: Illustrative examples from various vision-and-language datasets categorized by task type. Visual Question Answering (VQA) tasks are shown in red (VQAv2: train, VizWiz: laptop, OK-VQA: bus). Captioning tasks are represented in blue (Flickr30k: footbridge, MSCOCO: giraffes), while classification tasks are highlighted in green (HatefulMemes: meme identified as hateful). The bottom section demonstrates reasoning tasks with synthetic datasets: Fast Open-Ended MiniImageNet and CLEVR, focusing on conceptual understanding (e.g., assigning labels like "Dax" or identifying object properties like color and size).