If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selection

TL;DR

Large diffusion-based T2I models often struggle with faithful prompt rendering, but this paper shows that faithfulness can be significantly improved without retraining by selecting among multiple outputs. The proposed ImageSelect pipeline generates several candidate images from different seeds and ranks them with automatic faithfulness metrics (TIFA or ImageReward) to pick the best one. Across diverse prompts and benchmarks, ImageSelect yields substantial faithfulness gains and is supported by extensive human studies, often with similar or lower computational cost than methods that modify the generation process. This work offers a practical, scalable approach to improving text-to-image alignment and provides a strong reference for future post-hoc faithfulness enhancements.

Abstract

Despite their impressive capabilities, diffusion-based text-to-image (T2I) models can lack faithfulness to the text prompt, where generated images may not contain all the mentioned objects, attributes or relations. To alleviate these issues, recent works proposed post-hoc methods to improve model faithfulness without costly retraining, by modifying how the model utilizes the input prompt. In this work, we take a step back and show that large T2I diffusion models are more faithful than usually assumed, and can generate images faithful to even complex prompts without the need to manipulate the generative process. Based on that, we show how faithfulness can be simply treated as a candidate selection problem instead, and introduce a straightforward pipeline that generates candidate images for a text prompt and picks the best one according to an automatic scoring system that can leverage already existing T2I evaluation metrics. Quantitative comparisons alongside user studies on diverse benchmarks show consistently improved faithfulness over post-hoc enhancement methods, with comparable or lower computational cost. Code is available at \url{https://github.com/ExplainableML/ImageSelect}.

Figures (11)

• Figure 1: Our ImageSelect introduces automatic candidate selection to increase the faithfulness of a T2I generative model. We show that existing models are more faithful than assumed, and by simply querying them multiple times and selecting the most suitable image, we achieve significant improvements in T2I faithfulness, without requiring to explicitly adapt the generative process.
• Figure 2: Given a text prompt and a set of latent starting points $\epsilon_i$, we generate corresponding candidate images with off-the-shelf T2I models s.a. Stable Diffusion. A scoring mechanism then assigns faithfulness scores per image, with the highest scoring one simply selected as the final output.
• Figure 3: Quantitative results for baselines and ImageSelect on diverse-1k. For Stable Diffusion 1.4 and 2.1, ImageSelect outperforms all, irrespective of the selection and evaluation metric.
• Figure 4: RewardSelect offers improved faithfulness across faithfulness categories as used in hu2023tifa
• Figure 5: Faithfulness increases with number of candidate images per prompt to select from.