Evolutionary Prompt Optimization Discovers Emergent Multimodal Reasoning Strategies in Vision-Language Models

TL;DR

This work addresses the challenge of eliciting rich multimodal reasoning in vision-language systems without retraining. It introduces an inference-time evolutionary prompt optimization framework that evolves task prompts, mutation strategies, and meta-mutations across three hierarchical spaces, guided by a fitness function that combines task performance and LLM-based critique. A key finding is the emergence of tool-synthesis behaviors, where evolved prompts generate XML-tagged tool calls and downstream Python code to perform image operations, enabling multi-pass, hierarchical reasoning and significant performance gains across MathVista, M3CoT, and GeoBench-VLM benchmarks (up to approximately 50% relative improvement). The results underscore the potential of system prompts as neural programs and point toward native multimodal reasoning and omnimodal AI, while acknowledging limitations such as generalization across tasks and reliance on inference-time compute.

Abstract

We present a framework for optimizing prompts in vision-language models to elicit multimodal reasoning without model retraining. Using an evolutionary algorithm to guide prompt updates downstream of visual tasks, our approach improves upon baseline prompt-updating algorithms, which lack evolution-style "survival of the fittest" iteration. Crucially, we find this approach enables the language model to independently discover progressive problem-solving techniques across several evolution generations. For example, the model reasons that to "break down" visually complex spatial tasks, making a tool call to a Python interpreter to perform tasks (such as cropping, image segmentation, or saturation changes) would improve performance significantly. Our experimentation shows that explicitly evoking this "tool calling" call, via system-level XML $...\texttt{<tool>} ... \texttt{</tool>}...$ tags, can effectively flag Python interpreter access for the same language model to generate relevant programs, generating advanced multimodal functionality. This functionality can be crystallized into a system-level prompt that induces improved performance at inference time, and our experimentation suggests up to $\approx 50\%$ relative improvement across select visual tasks. Downstream performance is trained and evaluated across subtasks from MathVista, M3CoT, and GeoBench-VLM datasets. Importantly, our approach shows that evolutionary prompt optimization guides language models towards self-reasoning discoveries, which result in improved zero-shot generalization across tasks.

Figures (7)

• Figure 1: Generalization performance of various prompt optimization techniques on Damaged Building Count vision-heavy reasoning task
• Figure 2: A naive example of directly prompting an input image using the original dataset prompt. Note that the LLM misses one of the metallic shapes, leading to an incorrect conclusion. Given the visual complexity of the input image, Vision Language Models may struggle to accurately analyze subcomponents without further guidance.
• Figure 3: Walk through of an example where an initial prompt fails to elicit a correct answer, while a successful evolutionarily optimized prompt including a tool call (cropping) succeeds. Via the evolved prompt, the model elicits a tool call that crops the original image, allowing the LLM to better ingest the image's contents. With the improved quadrant division of visual analysis, the model is able to correctly answer the question.
• Figure 4: Baseline evolutionary prompt optimization method (Promptbreeder, Fernando et. al. 2023) fails to generalize to vision-language reasoning domains. We find this is because their instruction-following prompting for LLM mutation, hypermutation, and their initial universes are not suited for vision-language reasoning tasks.
• Figure 5: Our naive method outperforms baselines in the evolution process, due to significant improvements in mutation methods, our auxillary loss preventing significant and nonsensical deviations from current task prompts, and our initial universes of task prompts, mutation prompts, and hypermutation prompts, that are tuned specifically for image tasks.