Adaptive Prompt Elicitation for Text-to-Image Generation

TL;DR

The paper tackles the persistent misalignment between user intent and text‑to‑image outputs caused by ambiguous prompts and model idiosyncrasies. It introduces Adaptive Prompt Elicitation (APE), which inverts prompting by actively eliciting latent intent through visually grounded queries and compiling responses into optimized prompts within an information‑theoretic framework. Across two benchmarks (DesignBench and IDEA‑Bench) and a 128‑participant user study, APE achieves stronger alignment with fewer interaction steps, demonstrating significant efficiency gains and improved user agency. The work contributes a principled, transparent, and generalizable prompting paradigm that complements model improvements, with source code available for replication. Overall, APE shows how interactive, visual elicitation can better bridge human intent and generative models, potentially extending to other modalities and domains.

Abstract

Aligning text-to-image generation with user intent remains challenging, for users who provide ambiguous inputs and struggle with model idiosyncrasies. We propose Adaptive Prompt Elicitation (APE), a technique that adaptively asks visual queries to help users refine prompts without extensive writing. Our technical contribution is a formulation of interactive intent inference under an information-theoretic framework. APE represents latent intent as interpretable feature requirements using language model priors, adaptively generates visual queries, and compiles elicited requirements into effective prompts. Evaluation on IDEA-Bench and DesignBench shows that APE achieves stronger alignment with improved efficiency. A user study with challenging user-defined tasks demonstrates 19.8% higher alignment without workload overhead. Our work contributes a principled approach to prompting that, for general users, offers an effective and efficient complement to the prevailing prompt-based interaction paradigm with text-to-image models.

Figures (14)

• Figure 1: User interface design. The left area functions as feature specification, consisting of (A) question panel that presents visual queries, and (B) requirements panel that presents visual feature requirements specified during the interactions. (C) The user can generate an image by clicking the "Submit" button, which translates the requirements into effective prompts to generate an image.
• Figure 2: APE's computational pipeline. The system maintains a user intent model over visual features, adaptively generates visual queries to infer the user's intent, and then synthesizes optimized prompts for text-to-image generation models.
• Figure 3: The progress of alignment over iterations, evaluated by image-image similarity (DreamSim$\uparrow$) with 95% CI. APE demonstrates faster convergence and higher final alignment compared to In-Context Query.
• Figure 4: Alignment ratings (± SE) for four dimensions across conditions. Participants using APE reported significantly higher scores for all dimensions compared to Baseline. Asterisks indicate significance levels (* $p < 0.05$, ** $p < 0.01$, *** $p < 0.001$).
• Figure 5: Alignment ratings (sum $\pm$ SD) by condition and prior experience level.