Text-to-Image Diffusion Models Cannot Count, and Prompt Refinement Cannot Help

TL;DR

This work introduces T2ICountBench, a rigorous benchmark to evaluate the counting ability of text-to-image diffusion models by isolating numerical accuracy across a wide range of object counts ($1$ to $15$), object categories, scenes, and styles, with full human evaluation. Across 15 state-of-the-art models (open-source and private), counting accuracy remains poor and degrades as complexity increases, with best averages around $43\%$ and many models near or below $30\%$. An extensive ablation shows difficulty grows with object count and scene complexity, while style has a comparatively smaller effect. A prompt-refinement study shows that simple task-decomposition strategies largely fail to improve counting, suggesting intrinsic limitations in numerical understanding and alignment in current diffusion models and pointing to future directions in CLIP counting improvements, automatic prompt refinement, and human-preference alignment.

Abstract

Generative modeling is widely regarded as one of the most essential problems in today's AI community, with text-to-image generation having gained unprecedented real-world impacts. Among various approaches, diffusion models have achieved remarkable success and have become the de facto solution for text-to-image generation. However, despite their impressive performance, these models exhibit fundamental limitations in adhering to numerical constraints in user instructions, frequently generating images with an incorrect number of objects. While several prior works have mentioned this issue, a comprehensive and rigorous evaluation of this limitation remains lacking. To address this gap, we introduce T2ICountBench, a novel benchmark designed to rigorously evaluate the counting ability of state-of-the-art text-to-image diffusion models. Our benchmark encompasses a diverse set of generative models, including both open-source and private systems. It explicitly isolates counting performance from other capabilities, provides structured difficulty levels, and incorporates human evaluations to ensure high reliability. Extensive evaluations with T2ICountBench reveal that all state-of-the-art diffusion models fail to generate the correct number of objects, with accuracy dropping significantly as the number of objects increases. Additionally, an exploratory study on prompt refinement demonstrates that such simple interventions generally do not improve counting accuracy. Our findings highlight the inherent challenges in numerical understanding within diffusion models and point to promising directions for future improvements.

Figures (74)

• Figure 1: Impact of Difficulty Levels. This figure presents the comparison of the accuracy of various models across three difficulty levels (Easy, Medium, Hard). The horizontal axis lists the models, while the vertical axis represents accuracy. Each bar in the figure represents the accuracy for a specific model under the corresponding prompt difficulty level.
• Figure 2: Qualitative Study of Different Difficulty Levels. A high-resolution version of this image is available in Figure \ref{['fig:qualitative_2']} in Appendix \ref{['sec:append_qual_study']}.
• Figure 3: Impact of Style. This figure presents the comparison of the accuracy of various models across three styles (Plain, Watercolor, Cartoon). The horizontal axis lists the models, while the vertical axis represents accuracy. Each bar in the figure represents the accuracy for a specific model under corresponding prompt style setting.
• Figure 4: Impact of Scene. This figure presents the comparison of the accuracy of various models across three scenes (Home, Nature, City). The horizontal axis lists the models, while the vertical axis represents accuracy. Each bar in the figure represents the accuracy for a specific model under the corresponding prompt scene setting.
• Figure 5: Impact of Scene and Style on Average Accuracy. Left: This figure presents a comparison of the average accuracy across three scenes (Home, Nature, City) for 15 models. Each bar represents the average accuracy of the 15 models under the corresponding prompt scene setting. Right: This figure presents a comparison of the average accuracy across three styles (Plain, Watercolor, Cartoon) for 15 models. Each bar represents the average accuracy of the 15 models under the corresponding prompt style setting.