Enhancing Multimodal Large Language Models with Multi-instance Visual Prompt Generator for Visual Representation Enrichment

TL;DR

Multimodal Large Language Models often rely on single-image inputs and standard visual prompt generators, limiting their ability to leverage richer visual representations. This paper introduces the Multi-instance Visual Prompt Generator (MIVPG), a MIL-inspired module that aggregates across images and patches and includes Correlated Self-Attention (CSA) to capture inter-instance relationships, with QFormer shown as a special case. Across three vision-language datasets spanning natural imagery, pathology WSIs, and e-commerce products, MIVPG consistently outperforms QFormer-based baselines, particularly in data-scarce regimes and multi-input scenarios. The work provides both theoretical and empirical support for MIL-based visual prompting and offers a practical path to richer multimodal understanding in LLMs.

Abstract

Multimodal Large Language Models (MLLMs) have achieved SOTA performance in various visual language tasks by fusing the visual representations with LLMs leveraging some visual adapters. In this paper, we first establish that adapters using query-based Transformers such as Q-former is a simplified Multi-instance Learning method without considering instance heterogeneity/correlation. We then propose a general component termed Multi-instance Visual Prompt Generator (MIVPG) to incorporate enriched visual representations into LLMs by taking advantage of instance correlation between images or patches for the same sample. Quantatitive evaluation on three public vision-language (VL) datasets from different scenarios shows that the proposed MIVPG improves Q-former in main VL tasks.

Figures (18)

• Figure 1: Left: Exemplary images from collins2022abo, portraying e-commerce products captured from various aspects. Right: Illustration of a Whole Slide Image (WSI) sourced from tsuneki2022inference. Each WSI is composed of multiple patches, exhibiting dimensions comparable to those of natural images.
• Figure 2: Overview of MIVPG. \ref{['fig:method_overview_1']}: When handling multiple visual inputs, the initial step involves aggregating them at the image-level. QFormer can be treated as a Multiple Instance Learning module that takes multiple samples as instances. The MIVPG complements QFormer by introducing a correlated self-attention module and the pyramid positional encoding module, depending on specific scenarios. \ref{['fig:method_overview_2']}: Image-level aggregation can employ various MIL strategies, either learnable, such as AB-MIL, or fixed, for example, always selecting a specific token. \ref{['fig:method_overview_3']}: The visual prompt embeddings produced by Q-Former are combined with textual prompt embeddings and forwarded to the LLM for generating outputs.
• Figure 3: Left: The original transformer block without considering instance correlation. Middle: Instance correlation is computed through a self-attention layer among input instances, incurring a time complexity of $\mathcal{O}(M^2)$. Right: Instance correlation is calculated using query embeddings from the previous layer. This approach reduces the time complexity in computing correlation to $\mathcal{O}(MR)$.
• Figure 4: Experiment Results on MSCOCO. We adopt the metrics used in li2023blip. It is evident that the incorporation of MIL modules enhances the QFormer in the majority of cases.
• Figure 5: Visualization of Inference Results on PatchGastricADC22. We highlight details that should be focused on the reference. Zero-shot inference is performed using the pretrained BLIP2li2023blip. As the number of epochs increases, the model acquires more domain knowledge.

Theorems & Definitions (3)

• Proposition 1
• Proposition 2
• proof