Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs

TL;DR

This work introduces multimodal prompt optimization to extend beyond text-only prompts for Multimodal LLMs (MLLMs). It presents MPO, a framework that jointly updates textual and non-textual prompts in an alignment-preserving manner and uses prior-informed Bayesian UCB to efficiently select high-performing prompts. Across image, video, and molecular tasks, MPO consistently outperforms text-only prompt optimizers and manual prompts, demonstrating the importance of leveraging cross-modal signals. The approach delivers strong generalization across backbones, reveals a positive correlation between parent and child prompt performance, and shows substantial reductions in evaluation budgets while enhancing multimodal reasoning capabilities. Overall, MPO establishes multimodal prompt optimization as a key step toward fully exploiting MLLMs' multimodal capacity.

Abstract

Large Language Models (LLMs) have shown remarkable success, and their multimodal expansions (MLLMs) further unlock capabilities spanning images, videos, and other modalities beyond text. However, despite this shift, prompt optimization approaches, designed to reduce the burden of manual prompt crafting while maximizing performance, remain confined to text, ultimately limiting the full potential of MLLMs. Motivated by this gap, we introduce the new problem of multimodal prompt optimization, which expands the prior definition of prompt optimization to the multimodal space defined by the pairs of textual and non-textual prompts. To tackle this problem, we then propose the Multimodal Prompt Optimizer (MPO), a unified framework that not only performs the joint optimization of multimodal prompts through alignment-preserving updates but also guides the selection process of candidate prompts by leveraging earlier evaluations as priors in a Bayesian-based selection strategy. Through extensive experiments across diverse modalities that go beyond text, such as images, videos, and even molecules, we demonstrate that MPO outperforms leading text-only optimization methods, establishing multimodal prompt optimization as a crucial step to realizing the potential of MLLMs.

Figures (16)

• Figure 1: Concept Figure. (A) Existing prompt optimization approaches restrict the optimization to the textual space, leaving MLLMs underutilized by failing to provide rich contextual signals. (B) Our multimodal prompt optimization expands the optimization space into multimodality, allowing the discovery of salient multimodal context and fully leveraging the expressive capacity of MLLMs.
• Figure 2: Overview of MPO, consisting of two components. (A) Alignment-preserving exploration analyzes a failure set to generate feedback, which is then used both to refine the textual prompt and to guide a modality-specific generator to create a new non-textual prompt with one of three operators. (B) Prior-Inherited Bayesian UCB Selection leverages the parent's performance as an informative prior, warm-starting the search to effectively identify high-performing prompts among candidates.
• Figure 2: Generalizability results of MPO across components with different backbones: (Top) base models; (Bottom Left) optimizer models; (Bottom Right) modality-specific generators.
• Figure 3: Correlation of parent and child scores.
• Figure 3: Ablation on the contribution of each modality in the optimized multimodal prompt.

Theorems & Definitions (4)

• Proposition 3.1
• Lemma B.1: Pseudo-counts shrink one-sided credible widths
• Lemma B.2: Effect of Informative Priors on Posterior Quantiles
• proof