Can You Count to Nine? A Human Evaluation Benchmark for Counting Limits in Modern Text-to-Video Models

TL;DR

This work introduces T2VCountBench, a human-evaluated benchmark to probe counting adherence in modern text-to-video diffusion models as of 2025. Across 10 generators and 165 prompts varying object, scene, and motion factors, all models fail to reliably count objects up to $9$, despite generally high fidelity. Ablation studies show counting accuracy degrades sharply with higher counts, while video style, temporal dynamics, and multilingual prompts only modestly influence performance; simple prompt refinement often yields little or negative benefit. The findings highlight a critical gap in current text-to-video systems: reliable enforcement of basic numerical constraints, motivating future work on numerically faithful generation for safer and more trustworthy AI video synthesis.

Abstract

Generative models have driven significant progress in a variety of AI tasks, including text-to-video generation, where models like Video LDM and Stable Video Diffusion can produce realistic, movie-level videos from textual instructions. Despite these advances, current text-to-video models still face fundamental challenges in reliably following human commands, particularly in adhering to simple numerical constraints. In this work, we present T2VCountBench, a specialized benchmark aiming at evaluating the counting capability of SOTA text-to-video models as of 2025. Our benchmark employs rigorous human evaluations to measure the number of generated objects and covers a diverse range of generators, covering both open-source and commercial models. Extensive experiments reveal that all existing models struggle with basic numerical tasks, almost always failing to generate videos with an object count of 9 or fewer. Furthermore, our comprehensive ablation studies explore how factors like video style, temporal dynamics, and multilingual inputs may influence counting performance. We also explore prompt refinement techniques and demonstrate that decomposing the task into smaller subtasks does not easily alleviate these limitations. Our findings highlight important challenges in current text-to-video generation and provide insights for future research aimed at improving adherence to basic numerical constraints.

Figures (50)

• Figure 1: Impact of Scene Transition on Counting Accuracy.
• Figure 2: Impact of Multilingual Prompts on Counting Accuracy.
• Figure 3: Qualitative Study on Multilingual Prompts. The Spanish prompt describes "five girls eating burgers", while the Chinese prompt describes "nine butterflies flying".
• Figure 4: Impact of Prompt Refinement on Counting Accuracy.
• Figure 5: Impact of Prompt Refinement on Object Fidelity.