D2D: Detector-to-Differentiable Critic for Improved Numeracy in Text-to-Image Generation

TL;DR

The paper tackles numeracy in text-to-image diffusion by leveraging non-differentiable detectors as counting priors. It introduces D2D, which converts detectors into differentiable critics via a high-curvature activation and guides initial-noise optimization at inference with a Latent Modifier Network. Across multiple backbones and datasets, D2D achieves substantial improvements in object-count accuracy while maintaining image quality and modest compute overhead. The approach broadens numeracy capabilities by harnessing strong open-vocabulary detectors and a lightweight, generalizable optimization framework. This has practical impact for rendering images with precise object counts in open-world scenarios.

Abstract

Text-to-image (T2I) diffusion models have achieved strong performance in semantic alignment, yet they still struggle with generating the correct number of objects specified in prompts. Existing approaches typically incorporate auxiliary counting networks as external critics to enhance numeracy. However, since these critics must provide gradient guidance during generation, they are restricted to regression-based models that are inherently differentiable, thus excluding detector-based models with superior counting ability, whose count-via-enumeration nature is non-differentiable. To overcome this limitation, we propose Detector-to-Differentiable (D2D), a novel framework that transforms non-differentiable detection models into differentiable critics, thereby leveraging their superior counting ability to guide numeracy generation. Specifically, we design custom activation functions to convert detector logits into soft binary indicators, which are then used to optimize the noise prior at inference time with pre-trained T2I models. Our extensive experiments on SDXL-Turbo, SD-Turbo, and Pixart-DMD across four benchmarks of varying complexity (low-density, high-density, and multi-object scenarios) demonstrate consistent and substantial improvements in object counting accuracy (e.g., boosting up to 13.7% on D2D-Small, a 400-prompt, low-density benchmark), with minimal degradation in overall image quality and computational overhead.

Figures (16)

• Figure 1: Qualitative examples illustrating the count-correction ability of our detector-based critic on a variety of objects, counts 1-10. SDXL-Turbo sauer2025turbo is a base model with no post-enhancement. ReNO eyring2024reno is a generic semantic alignment method that exhibits limited performance in this setting. More recent methods, like Make It Count binyamin2025mic and Counting Guidance kang2025cg, explicitly address count-correction. Our method proposes a new and effective way to leverage detectors for this challenging task. Prompt template: "A realistic photo of a scene with [count] [object class]."
• Figure 2: The low-density setting is where incorrect numeracy is most noticeable and also where detectors count better than regression-based methods.But detectors are not differentiable, which precludes them from being used as critics for count correction.
• Figure 3: The D2D pipeline for improving T2I numeracy.D2D consists of two main components that work together to improve numeracy: our detector-based count critic guides the Latent Modifier Network (LMN) on how to transform the original initial noise $\mathbf{x}_T$ into a more optimal $\mathbf{x}_T'$. Our count critic uses sigmoid-based activation functions to convert logits into gradient signals, which are then backpropagated through the frozen generator to update the weights of the LMN.
• Figure 4: D2D improves numeracy on the majority of the 41 objects in CoCoCount and D2D-Small. Evaluated against ReNO eyring2024reno and TokenOpt zafar2024tokenopt on base SDXL-Turbo. Avg. over four seeds.
• Figure 5: D2D effectively corrects over and under-generation. The initial generation contains six more dogs/one fewer cup than requested, which our method iteratively corrects, arriving at an image of 10 dogs/four cups, as requested.