IPO: Interpretable Prompt Optimization for Vision-Language Models

TL;DR

A Prompt Optimization Prompt that not only guides LLMs in creating effective prompts but also stores past prompts with their performance metrics, providing rich in-context information is introduced, which ensures that the prompts remain human-understandable.

Abstract

Pre-trained vision-language models like CLIP have remarkably adapted to various downstream tasks. Nonetheless, their performance heavily depends on the specificity of the input text prompts, which requires skillful prompt template engineering. Instead, current approaches to prompt optimization learn the prompts through gradient descent, where the prompts are treated as adjustable parameters. However, these methods tend to lead to overfitting of the base classes seen during training and produce prompts that are no longer understandable by humans. This paper introduces a simple but interpretable prompt optimizer (IPO), that utilizes large language models (LLMs) to generate textual prompts dynamically. We introduce a Prompt Optimization Prompt that not only guides LLMs in creating effective prompts but also stores past prompts with their performance metrics, providing rich in-context information. Additionally, we incorporate a large multimodal model (LMM) to condition on visual content by generating image descriptions, which enhance the interaction between textual and visual modalities. This allows for thae creation of dataset-specific prompts that improve generalization performance, while maintaining human comprehension. Extensive testing across 11 datasets reveals that IPO not only improves the accuracy of existing gradient-descent-based prompt learning methods but also considerably enhances the interpretability of the generated prompts. By leveraging the strengths of LLMs, our approach ensures that the prompts remain human-understandable, thereby facilitating better transparency and oversight for vision-language models.

Figures (9)

• Figure 1: Comparison between traditional gradient-based prompt optimization (a) and our interpretable prompt optimization (b) for vision-language models. Traditional gradient descent-based prompt learning methods COOPcocoop treat the text prompt as learnable parameters $V$. By minimizing the loss through gradient descent on the training set, an optimized prompt $\hat{V}$ is obtained after $I$ iterations, which is not interpretable by humans. In contrast, our interpretable prompt optimization leverages an LLM as optimizer to optimize the loss and accuracy. After $I$ iterations, the resulting optimized top prompt is effective and human-readable.
• Figure 2: An example of our Prompt Optimization Prompt with input and output on the DTD dtd dataset. The red text represents instructions given to the large language model, the blue text denotes the image descriptions generated by the large multimodal model, and the green text indicates the top-20 previously generated prompts retrieved from episodic memory along with their corresponding scores. yellow indicates the output prompt.
• Figure 3: An example of our Prompt Optimization Prompt with input with initial instruction on the Food101 food101 dataset.
• Figure 4: An example of our Prompt Optimization Prompt with input at step 1 on the Food101 food101 dataset.
• Figure 5: An example of our Prompt Optimization Prompt with input step 100 on the Food101 food101 dataset.