Reverse Prompt: Cracking the Recipe Inside Text-to-Image Generation

TL;DR

This work addresses the challenge of decoding reverse prompts from reference images for text-to-image generation. It introduces Automatic Reverse Prompt Optimization (ARPO), a gradient-imitation, three-stage framework that initializes prompts, iteratively regenerates images, and uses two prompt-generation strategies (vanilla and enhanced) followed by a greedy CLIP-based prompt selection to maximize similarity to the reference. ARPO not only recreates reference-style images with high fidelity but also enables novel image generation by editing prompt components that separate content and style. Extensive quantitative and qualitative experiments demonstrate ARPO's superiority over gradient-based, captioning, and data-driven baselines across multiple T2I models, with open-source and closed-source variants offering different trade-offs between cost and speed. The work also emphasizes practical applications like image modification and fusion, and commits to releasing code and data to foster further research.

Abstract

Text-to-image generation has become increasingly popular, but achieving the desired images often requires extensive prompt engineering. In this paper, we explore how to decode textual prompts from reference images, a process we refer to as image reverse prompt engineering. This technique enables us to gain insights from reference images, understand the creative processes of great artists, and generate impressive new images. To address this challenge, we propose a method known as automatic reverse prompt optimization (ARPO). Specifically, our method refines an initial prompt into a high-quality prompt through an iteratively imitative gradient prompt optimization process: 1) generating a recreated image from the current prompt to instantiate its guidance capability; 2) producing textual gradients, which are candidate prompts intended to reduce the difference between the recreated image and the reference image; 3) updating the current prompt with textual gradients using a greedy search method to maximize the CLIP similarity between prompt and reference image. We compare ARPO with several baseline methods, including handcrafted techniques, gradient-based prompt tuning methods, image captioning, and data-driven selection method. Both quantitative and qualitative results demonstrate that our ARPO converges quickly to generate high-quality reverse prompts. More importantly, we can easily create novel images with diverse styles and content by directly editing these reverse prompts. Code will be made publicly available.

Figures (29)

• Figure 1: Illustration of image reverse prompt engineering. Given a reference image, the goal is to identify a reverse prompt that effectively recreates the image with similar content and style.
• Figure 2: Illustration of recreated images using different reverse prompts.
• Figure 3: The main ARPO framework consists of three main components: image generation, prompt generation, and prompt selection.
• Figure 4: Two prompt generation frameworks used in ARPO. (a) Vanilla prompt generation. (b) Enhanced prompt generation.
• Figure 5: Illustration of novel image generation. More details are provided in Appendix \ref{['appendix:novel_generation_details']}.