The Intricate Dance of Prompt Complexity, Quality, Diversity, and Consistency in T2I Models

TL;DR

The paper investigates how prompt complexity affects the utility of synthetic data produced by text-to-image models, focusing on quality, diversity, and consistency. It introduces a benchmarking framework and validates it with synthetic toy experiments and large-scale evaluations across CC12M, ImageNet-1k, and DCI, using multiple T2I models and inference-time interventions such as prompt expansion and advanced guidance. Key findings show that generalizing to very general prompts is hard for diffusion-based generators, prompting a trade-off where increasing prompt complexity reduces diversity and consistency but lowers the synthetic-real distribution gap; prompt expansion with a pretrained language model often yields the highest diversity and aesthetics, sometimes exceeding real data, with further gains when combined with APG. The work provides actionable guidance for constructing prompts and selecting inference-time interventions to maximize the utility of synthetic data, while highlighting that diversity remains a distinguishing factor of real-world data distributions.

Abstract

Text-to-image (T2I) models offer great potential for creating virtually limitless synthetic data, a valuable resource compared to fixed and finite real datasets. Previous works evaluate the utility of synthetic data from T2I models on three key desiderata: quality, diversity, and consistency. While prompt engineering is the primary means of interacting with T2I models, the systematic impact of prompt complexity on these critical utility axes remains underexplored. In this paper, we first conduct synthetic experiments to motivate the difficulty of generalization w.r.t. prompt complexity and explain the observed difficulty with theoretical derivations. Then, we introduce a new evaluation framework that can compare the utility of real data and synthetic data, and present a comprehensive analysis of how prompt complexity influences the utility of synthetic data generated by commonly used T2I models. We conduct our study across diverse datasets, including CC12M, ImageNet-1k, and DCI, and evaluate different inference-time intervention methods. Our synthetic experiments show that generalizing to more general conditions is harder than the other way round, since the former needs an estimated likelihood that is not learned by diffusion models. Our large-scale empirical experiments reveal that increasing prompt complexity results in lower conditional diversity and prompt consistency, while reducing the synthetic-to-real distribution shift, which aligns with the synthetic experiments. Moreover, current inference-time interventions can augment the diversity of the generations at the expense of moving outside the support of real data. Among those interventions, prompt expansion, by deliberately using a pre-trained language model as a likelihood estimator, consistently achieves the highest performance in both image diversity and aesthetics, even higher than that of real data.

Figures (28)

• Figure 1: Generalization to prompts of different complexities during inference. \ref{['fig:synthetic_training']} shows the training data distribution. \ref{['fig:synthetic_general_cat']} presents the generated samples using the general prompt cat with the model trained with fine-grained prompts. \ref{['fig:synthetic_black_dog']} shows the generated samples using the fine-grained prompt black dog with the model trained with general prompts. $\omega$ is the classifier-free guidance scale. With $\omega>1$, generalization towards more general prompts is harder in this synthetic setting.
• Figure 2: Reference-free diversity metric. Diversity (Vendi) of LDMv1.5 and LDMv3.5L generations with CC12M and ImageNet-1k prompts when using: 1) vanilla guidance (CFG), 2) prompt expansion, and 3) advanced guidance methods, for which transparent markers correspond to different methods and the solid marker is the average over methods. Both advanced guidance methods and prompt expansion lead to improved diversity over the vanilla guidance. Prompt expansion from shorter captions can surpass the real data diversity. We further extend to much longer DCI prompts. Diversity of all models first decreases then plateaus which is not observed within shorter prompt length ranges.
• Figure 3: Reference-free quality metric. Quality (aesthetic) of LDMv1.5 and LDMv3.5L generations with CC12M and ImageNet-1k prompts when using: 1) vanilla guidance (CFG), 2) prompt expansion, and 3) advanced guidance methods, for which transparent markers correspond to different methods and the solid marker is the average over methods. Advanced guidance methods slightly impair the quality while prompt expansion increases it. We further extend to much longer DCI prompts. Quality of all models first increases then decreases which is not observed within shorter prompt ranges.
• Figure 4: Reference-free consistency metric. Consistency (DSG) metrics of LDMv1.5 and LDMv3.5L generations with CC12M and ImageNet-1k prompts when using: 1) vanilla guidance (CFG), 2) prompt expansion, and 3) advanced guidance methods, for which transparent markers correspond to different methods and the solid marker is the average over methods. Both advanced guidance methods and prompt expansion lead to lower consistency scores compared to vanilla guidance. We further extend to much longer DCI prompts. Consistency of all models decreases when the prompt lengths increases, which is the same as in the shorter prompt ranges.
• Figure 5: Reference-based utility metrics of synthetic data using CC12M prompts. FDD, precision, density and coverage for LDMv1.5 and LDMv3.5L generations with: (1) vanilla guidance (CFG), (2) prompt expansion, and (3) advanced guidance methods, for which transparent markers correspond to different methods and the solid marker is the average over methods. Both advanced guidance methods and prompt expansion lead to better FDD. Although prompt expansion improves coverage and advanced guidance methods match coverage for LDMv3.5, they both sacrifice precision and density. LDMv1.5 has thus better overall performance (lower FDD) than LDMv3.5L.