Vision-Driven Prompt Optimization for Large Language Models in Multimodal Generative Tasks

TL;DR

VDPO addresses the challenge of integrating visual understanding with high-quality image generation by using LLMs to produce adaptive prompts derived from visual cues. It introduces a visual embedding prompt tuner and a dual-modality alignment objective to guide a diffusion-based generator, enabling context-aware, semantically rich image synthesis. Across synthetic and real-world benchmarks (e.g., COCO, Sketchy), VDPO achieves state-of-the-art FID, LPIPS, and BLEU/CIDEr scores, with strong in-domain and out-of-domain generalization and favorable human evaluations. This approach advances multimodal generation by reducing dependence on handcrafted prompts and demonstrating scalable, robust generation through a learnable prompt optimization loop.

Abstract

Vision generation remains a challenging frontier in artificial intelligence, requiring seamless integration of visual understanding and generative capabilities. In this paper, we propose a novel framework, Vision-Driven Prompt Optimization (VDPO), that leverages Large Language Models (LLMs) to dynamically generate textual prompts from visual inputs, guiding high-fidelity image synthesis. VDPO combines a visual embedding prompt tuner, a textual instruction generator, and a vision generation module to achieve state-of-the-art performance in diverse vision generation tasks. Extensive experiments on benchmarks such as COCO and Sketchy demonstrate that VDPO consistently outperforms existing methods, achieving significant improvements in FID, LPIPS, and BLEU/CIDEr scores. Additional analyses reveal the scalability, robustness, and generalization capabilities of VDPO, making it a versatile solution for in-domain and out-of-domain tasks. Human evaluations further validate the practical superiority of VDPO in generating visually appealing and semantically coherent outputs.