Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledge

TL;DR

This work tackles the challenge of conveying fine-grained, spatially dense visual information to multimodal large language models. It introduces a visual prompt paradigm that embeds external knowledge from panoptic segmentation and OCR directly into a pixel-wise visual prompt, which is then infused with image features in a pixel-wise manner. Through LoRA-based fine-tuning on existing LLaVA-1.5 data, the approach yields consistent gains across nine benchmarks, often surpassing larger models while requiring no additional pre-training data. The results demonstrate the practical value of external, region-level knowledge for improving fine-grained context understanding in MLLMs, with implications for more reliable visual reasoning and reduced hallucinations in multimodal tasks.

Abstract

In recent years, multimodal large language models (MLLMs) have made significant strides by training on vast high-quality image-text datasets, enabling them to generally understand images well. However, the inherent difficulty in explicitly conveying fine-grained or spatially dense information in text, such as masks, poses a challenge for MLLMs, limiting their ability to answer questions requiring an understanding of detailed or localized visual elements. Drawing inspiration from the Retrieval-Augmented Generation (RAG) concept, this paper proposes a new visual prompt approach to integrate fine-grained external knowledge, gleaned from specialized vision models (e.g., instance segmentation/OCR models), into MLLMs. This is a promising yet underexplored direction for enhancing MLLMs' performance. Our approach diverges from concurrent works, which transform external knowledge into additional text prompts, necessitating the model to indirectly learn the correspondence between visual content and text coordinates. Instead, we propose embedding fine-grained knowledge information directly into a spatial embedding map as a visual prompt. This design can be effortlessly incorporated into various MLLMs, such as LLaVA and Mipha, considerably improving their visual understanding performance. Through rigorous experiments, we demonstrate that our method can enhance MLLM performance across nine benchmarks, amplifying their fine-grained context-aware capabilities.

Figures (3)

• Figure 1: Different training paradigms. (a) means the original visual instruction tuning of LLaVA liu2023improved. (b) denotes visual instruction tuning with external textual prompts jiao2024enhancing (e.g., 1 person and the center coordinates of its bounding box: [0.55,0.49]), note that we neglect the template prefix of textual prompts for visualization. (c) is the proposed auxiliary visual prompt, which is a feature map composed with different object regions. For each pixel, it is filled out with the textual embedding of the corresponding categories or OCR text ($t_{\mathrm{g}}$, $t_{\mathrm{p}}$ and $t_{\mathrm{b}}$ in the example visual prompt mean the textual embeddings of grass, person and baseball glove).
• Figure 2: Auxiliary visual prompt generation. It firstly generates the panoptic segmentation masks zhang2023simple for the input image, there's a class category for each mask region, then we can obtain the textual embeddings (e.g., $t_{\mathrm{book}}$, $t_{\mathrm{bed}}$ and $t_{\mathrm{cat}}$) through a pre-trained text encoder for all the classes (e.g., book, bed, cat). Finally, the auxiliary visual prompt can be generated by concatenating these textual embeddings within the corresponding mask regions together. Note that we can also adopt the OCR model du2021pp to obtain the texts and the regions, we don't display it here for clearer explanation.
• Figure 3: The illustration of visual instruction tuning with the generated visual prompt. Our proposed visual prompt can be easily combined with existing multimodal large language models (e.g., LLaVA liu2023improved), note that PEN means prompt embedding network.