Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Visual Prompting in Multimodal Large Language Models: A Survey

Junda Wu, Zhehao Zhang, Yu Xia, Xintong Li, Zhaoyang Xia, Aaron Chang, Tong Yu, Sungchul Kim, Ryan A. Rossi, Ruiyi Zhang, Subrata Mitra, Dimitris N. Metaxas, Lina Yao, Jingbo Shang, Julian McAuley

TL;DR

This survey maps the emergence of visual prompting for multimodal large language models, detailing a four-part taxonomy (Bounding-Box, Markers, Pixel-level, Soft prompts) and a generation pipeline (engineering, segmentation, detection, toolchains, learnable prompts). It analyzes how prompts integrate with visual encoders to improve grounding, object referring, and compositional reasoning, and covers training and in-context learning approaches that align MLLMs with multimodal instructions. The paper also reviews frontier applications, including safety, hallucination mitigation, and debiasing, and discusses evaluation benchmarks and datasets. By organizing methods, benchmarks, and design considerations, it provides a roadmap for robust, controllable multimodal reasoning in current and future MLLMs.

Abstract

Multimodal large language models (MLLMs) equip pre-trained large-language models (LLMs) with visual capabilities. While textual prompting in LLMs has been widely studied, visual prompting has emerged for more fine-grained and free-form visual instructions. This paper presents the first comprehensive survey on visual prompting methods in MLLMs, focusing on visual prompting, prompt generation, compositional reasoning, and prompt learning. We categorize existing visual prompts and discuss generative methods for automatic prompt annotations on the images. We also examine visual prompting methods that enable better alignment between visual encoders and backbone LLMs, concerning MLLM's visual grounding, object referring, and compositional reasoning abilities. In addition, we provide a summary of model training and in-context learning methods to improve MLLM's perception and understanding of visual prompts. This paper examines visual prompting methods developed in MLLMs and provides a vision of the future of these methods.

Visual Prompting in Multimodal Large Language Models: A Survey

TL;DR

This survey maps the emergence of visual prompting for multimodal large language models, detailing a four-part taxonomy (Bounding-Box, Markers, Pixel-level, Soft prompts) and a generation pipeline (engineering, segmentation, detection, toolchains, learnable prompts). It analyzes how prompts integrate with visual encoders to improve grounding, object referring, and compositional reasoning, and covers training and in-context learning approaches that align MLLMs with multimodal instructions. The paper also reviews frontier applications, including safety, hallucination mitigation, and debiasing, and discusses evaluation benchmarks and datasets. By organizing methods, benchmarks, and design considerations, it provides a roadmap for robust, controllable multimodal reasoning in current and future MLLMs.

Abstract

Multimodal large language models (MLLMs) equip pre-trained large-language models (LLMs) with visual capabilities. While textual prompting in LLMs has been widely studied, visual prompting has emerged for more fine-grained and free-form visual instructions. This paper presents the first comprehensive survey on visual prompting methods in MLLMs, focusing on visual prompting, prompt generation, compositional reasoning, and prompt learning. We categorize existing visual prompts and discuss generative methods for automatic prompt annotations on the images. We also examine visual prompting methods that enable better alignment between visual encoders and backbone LLMs, concerning MLLM's visual grounding, object referring, and compositional reasoning abilities. In addition, we provide a summary of model training and in-context learning methods to improve MLLM's perception and understanding of visual prompts. This paper examines visual prompting methods developed in MLLMs and provides a vision of the future of these methods.
Paper Structure (32 sections, 1 figure, 1 table)

This paper contains 32 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Visual Prompt Categorization
  3. Bounding-Box
  4. Markers
  5. Pixel-level
  6. Soft Visual Prompt
  7. Visual Prompt Generation
  8. Prompt Engineering
  9. Visual Segmentation
  10. Object Detection
  11. Visual Prompt Toolchain
  12. Learnable and Soft Visual Prompt
  13. Visual Perception
  14. Visual Grounding and Referring
  15. Multi-image and Video Understanding
  16. ...and 17 more sections

Figures (1)

  • Figure 1: Taxonomy flow chart of visual prompting techniques. We illustrate in order of four stages of visual prompting including prompt generation, visual prompting, perception and reasoning, and prompt learning, where the solid arrows show the direction of each component's information flow. We explain in detail various visual prompt generation techniques (Section \ref{['sec:gen']}), and how these generated prompts are used to prompt MLLMs (Section \ref{['sec:type']}). Then we discuss the advanced perception and reasoning abilities achieved through visual prompting (Section \ref{['sec:perception']} and \ref{['sec:reason']}). Finally, model pre-training, fine-tuning, instruction tuning, and in-context learning further update previous model components, which are illustrated by the dashed arrows (Section \ref{['sec:train']} and \ref{['sec:icl']}).