Demystifying Numerosity in Diffusion Models -- Limitations and Remedies

TL;DR

This work reveals that scaling diffusion models and data does not inherently fix numerosity in text-to-image generation. By constructing two synthetic benchmarks, GrayCount250 and NaturalCount6, the authors show that diffusion models rely more on noise priors than on textual prompts to determine object counts, leading to systematic counting errors. They demonstrate that noise priors bias both layout and count, and propose count-aware noise conditioning as an effective remedy, yielding substantial gains (e.g., 20.0% to 85.3% exact accuracy on GrayCount250 and 74.8% to 86.3% on NaturalCount6). The results generalize across controlled and naturalistic settings and offer practical guidance for improving numerosity control in diffusion models, challenging the assumption that larger scale alone resolves counting challenges.

Abstract

Numerosity remains a challenge for state-of-the-art text-to-image generation models like FLUX and GPT-4o, which often fail to accurately follow counting instructions in text prompts. In this paper, we aim to study a fundamental yet often overlooked question: Can diffusion models inherently generate the correct number of objects specified by a textual prompt simply by scaling up the dataset and model size? To enable rigorous and reproducible evaluation, we construct a clean synthetic numerosity benchmark comprising two complementary datasets: GrayCount250 for controlled scaling studies, and NaturalCount6 featuring complex naturalistic scenes. Second, we empirically show that the scaling hypothesis does not hold: larger models and datasets alone fail to improve counting accuracy on our benchmark. Our analysis identifies a key reason: diffusion models tend to rely heavily on the noise initialization rather than the explicit numerosity specified in the prompt. We observe that noise priors exhibit biases toward specific object counts. In addition, we propose an effective strategy for controlling numerosity by injecting count-aware layout information into the noise prior. Our method achieves significant gains, improving accuracy on GrayCount250 from 20.0\% to 85.3\% and on NaturalCount6 from 74.8\% to 86.3\%, demonstrating effective generalization across settings.

Figures (25)

• Figure 1: Key Motivations: On the left, we display typical images generated with the latest text-to-image models, most of which fail to adhere to numerosity instructions. On the right, we illustrate how numerosity poses a challenge for both text-to-image generation models and vision-language models.
• Figure 2: Pipeline for Constructing the GrayCount250 Dataset. We first prepare a list of concepts, and combine these concept names with a transparent object prompt for the FLUX.1-[dev]. Then, we apply RMBG 2.0 to perform matting and obtain the transparent object layers. Synthetic numerosity images are generated by pasting transparent object layers onto random layouts corresponding to target counts.
• Figure 3: Illustration of the transparent object layers used to construct the GrayCount250 dataset. All layers are rendered in a minimalist black line art style for simplicity.
• Figure 4: Illustration of the synthetic numerosity dataset. GrayCount250 (left): We show randomly arranged images containing 8 koalas, 15 pandas, 38 crocodiles, and 43 sea lions. NaturalCount6 (right): We show images with 3 avocados, 4 soccer balls, 5 wallets, and 6 remote controls with prompts provided in Appendix \ref{['appendix:prompt_templates']}.
• Figure 5: Our proposed coarse-to-fine numerosity classifier. Given an input image $x$ and a set of possible counts $k \in \{1,2,...,50\}$, we choose the conditioning $c$ ("A photo of $k$ rabbits.") that best predicts the noise added to the input image. To balance efficiency and accuracy, we employ a two-phase strategy: a Coarse Screening phase evaluates all 50 counts with 50 timesteps to identify the top 20 candidates, followed by a Refinement phase that uses 200 timesteps on these candidates to determine the final count.