VisualPrompter: Semantic-Aware Prompt Optimization with Visual Feedback for Text-to-Image Synthesis

TL;DR

This work proposes VisualPrompter, a novel training-free prompt engineering framework that refines user inputs to model-preferred sentences, and features a plug-and-play design, making it highly adaptable to various generative models.

Abstract

The notable gap between user-provided and model-preferred prompts poses a significant challenge for generating high-quality images with text-to-image models, compelling the need for prompt engineering. Current studies on prompt engineering can effectively enhance the style and aesthetics of generated images. However, they often neglect the semantic alignment between generated images and user descriptions, resulting in visually appealing but content-wise unsatisfying outputs. In this work, we propose VisualPrompter, a novel training-free prompt engineering framework that refines user inputs to model-preferred sentences. VisualPrompter utilizes an automatic self-reflection module that identifies absent concepts in the generated images, followed by a target-specific prompt optimization mechanism that revises the prompts in a fine-grained manner. By deconstructing prompts, introducing new elements at the atomic semantic level, and then reassembling them, our framework is able to maintain semantic consistency and integrity throughout the optimization process. Extensive experiments demonstrate the effectiveness of VisualPrompter, which achieves new state-of-the-art performance on multiple benchmarks for text-image alignment evaluation. Additionally, our framework features a plug-and-play design, making it highly adaptable to various generative models. Our code is available at https://github.com/teheperinko541/VisualPrompter.

Figures (17)

• Figure 1: Existing T2I generative models often fail to correctly draw key concepts within the provided prompts. Different models also fail differently on the same prompt.
• Figure 2: The refinement pipeline of our VisualPrompter. The optimization starts with LLM-driven VLM-verified semantic evaluation, followed by LLM-controlled concept expansion and sentence composition. Our approach emulates the chain-of-thought reasoning in human prompt refinement. All utilized models do not require any extra training.
• Figure 3: Comparison of prompts generated by large language model and different prompt engineering methods. Images in the two rows correspond to outputs from Stable Diffusion v2.1 and Flux-dev, respectively.
• Figure 4: Human evaluation results. From left to right, the segments indicate preference for images from optimized prompts, equal preference for both results, and preference for images from original prompts.
• Figure 5: Adaptation for online generators. The second row presents results optimized by VisualPrompter.