PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Prediction

TL;DR

PQPP delivers the first joint benchmark for prompt and query performance prediction in text-to-image generation and retrieval, compiling 10,200 prompts with extensive human judgments across generation and retrieval tasks. The dataset, built from MS COCO captions and DrawBench prompts, uses two diffusion models and two vision-language retrievers to produce a rich, multimodal evaluation resource. Across pre- and post-prediction paradigms, supervised predictors (notably fine-tuned CLIP and BERT) outperform basic features, though generation and retrieval remain only moderately correlated, highlighting task-specific challenges. PQPP enables cross-model, cross-dataset, and cross-task analyses and offers practical use cases for model selection and automatic prompt reformulation, with code and data publicly available for the research community.

Abstract

Text-to-image generation has recently emerged as a viable alternative to text-to-image retrieval, driven by the visually impressive results of generative diffusion models. Although query performance prediction is an active research topic in information retrieval, to the best of our knowledge, there is no prior study that analyzes the difficulty of queries (referred to as prompts) in text-to-image generation, based on human judgments. To this end, we introduce the first dataset of prompts which are manually annotated in terms of image generation performance. Additionally, we extend these evaluations to text-to-image retrieval by collecting manual annotations that represent retrieval performance. We thus establish the first joint benchmark for prompt and query performance prediction (PQPP) across both tasks, comprising over 10K queries. Our benchmark enables (i) the comparative assessment of prompt/query difficulty in both image generation and image retrieval, and (ii) the evaluation of prompt/query performance predictors addressing both generation and retrieval. We evaluate several pre- and post-generation/retrieval performance predictors, thus providing competitive baselines for future research. Our benchmark and code are publicly available at https://github.com/Eduard6421/PQPP.

Figures (9)

• Figure 1: We select a set of 10K captions from MS COCO Lin-ECCV-2014 using k-means clustering, which are further merged with prompts from DrawBench Saharia-NeurIPS-2022. Next, we collect human relevance judgments in two scenarios: image generation and image retrieval. For each prompt/query, we generate images with two diffusion models (Stable Diffusion XL Podell-ICLR-2024 and GLIDE Nichol-ICML-2021) and retrieve images from MS COCO with two vision-language models (CLIP Radford-ICML-2021 and BLIP-2 Li-ICML-2023). Based on the collected relevance judgments, we score each prompt/query in terms of generation and retrieval performance, respectively. Finally, we train and evaluate multiple prompt/query performance predictors on the proposed benchmark. Best viewed in color.
• Figure 2: An example showing the annotation interface for a random prompt and the images associated with the respective prompt. For each image, the annotator can select one of the following options: high relevance, low relevance, no relevance and unrealistic. The relevance judgments of an annotator are shown for illustrative purposes. The locations of images are randomly generated, each time they are displayed. Hence, the annotators do not see the images in the same order, which eliminates positional biases. Best viewed in color.
• Figure 3: Representative prompts/queries that exhibit high or low performance in text-to-image generation (on the horizontal axis) and text-to-image retrieval (on the vertical axis).
• Figure 4: t-SNE visualization of the test prompts embedded in the latent space of the BERT predictor fine-tuned on image generation with GLIDE. The ground-truth HBPP performance is encoded via a color map from green (high) to red (low). The visualization confirms that the fine-tuned BERT predictor learns a meaningful representation of the prompts. Best viewed in color.
• Figure 5: Predicted performance scores by a fine-tuned BERT model for GLIDE and SDXL on various test prompts. The scores are presented alongside the images generated by each model. Best viewed in color.