EDITOR: Effective and Interpretable Prompt Inversion for Text-to-Image Diffusion Models

TL;DR

The paper addresses the challenge ofPrompt inversion for text-to-image diffusion models by proposing EDITOR, a three-stage pipeline that initializes latent embeddings with an image captioning model, refines them through reverse-engineering in the latent space, and converts refined embeddings to text via an embedding-to-text decoder. It demonstrates superior image similarity, textual alignment, and interpretability over existing methods on MS COCO, LAION, and Flickr, and shows robustness across single- and multi-encoder diffusion models. The work enables practical downstream applications such as cross-concept synthesis, concept manipulation, evolutionary multi-concept generation, and unsupervised segmentation, while acknowledging ethical considerations and the potential for prompt IP leakage. Overall, EDITOR advances prompt inversion by maintaining semantic continuity in continuous latent space while yielding human-readable prompts with strong alignment to the target images.

Abstract

Text-to-image generation models~(e.g., Stable Diffusion) have achieved significant advancements, enabling the creation of high-quality and realistic images based on textual descriptions. Prompt inversion, the task of identifying the textual prompt used to generate a specific artifact, holds significant potential for applications including data attribution, model provenance, and watermarking validation. Recent studies introduced a delayed projection scheme to optimize for prompts representative of the vocabulary space, though challenges in semantic fluency and efficiency remain. Advanced image captioning models or visual large language models can generate highly interpretable prompts, but they often lack in image similarity. In this paper, we propose a prompt inversion technique called \sys for text-to-image diffusion models, which includes initializing embeddings using a pre-trained image captioning model, refining them through reverse-engineering in the latent space, and converting them to texts using an embedding-to-text model. Our experiments on the widely-used datasets, such as MS COCO, LAION, and Flickr, show that our method outperforms existing methods in terms of image similarity, textual alignment, prompt interpretability and generalizability. We further illustrate the application of our generated prompts in tasks such as cross-concept image synthesis, concept manipulation, evolutionary multi-concept generation and unsupervised segmentation.

Figures (17)

• Figure 1: Original prompts and generated images along with inverted prompts.
• Figure 2: Existing works optimize the token embedding before transformer layer and project them to nearest token embeddings, often producing incoherent results; EDITOR optimizes the contextual embedding after transformer layer and converts the final optimized embeddings into prompts. The red arrow indicates the backward pass in gradient-based optimization.
• Figure 3: Overview of EDITOR. Our approach comprises three steps: ① initializing the latent embedding via a pre-trained image captioning model; ② refining the embedding through reverse-engineering; ③ mapping the refined embedding to text with an embedding-to-text model. This pipeline yields coherent prompts and streamlines optimization.
• Figure 4: Prompt Inversion for Text-to-Image Diffusion Model
• Figure 5: Application of cross-concept image synthesis.