Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Numbers Speak: Aligning Textual Numerals and Visual Instances in Text-to-Video Diffusion Models

Zhengyang Sun, Yu Chen, Xin Zhou, Xiaofan Li, Xiwu Chen, Dingkang Liang, Xiang Bai

Abstract

Text-to-video diffusion models have enabled open-ended video synthesis, but often struggle with generating the correct number of objects specified in a prompt. We introduce NUMINA , a training-free identify-then-guide framework for improved numerical alignment. NUMINA identifies prompt-layout inconsistencies by selecting discriminative self- and cross-attention heads to derive a countable latent layout. It then refines this layout conservatively and modulates cross-attention to guide regeneration. On the introduced CountBench, NUMINA improves counting accuracy by up to 7.4% on Wan2.1-1.3B, and by 4.9% and 5.5% on 5B and 14B models, respectively. Furthermore, CLIP alignment is improved while maintaining temporal consistency. These results demonstrate that structural guidance complements seed search and prompt enhancement, offering a practical path toward count-accurate text-to-video diffusion. The code is available at https://github.com/H-EmbodVis/NUMINA.

When Numbers Speak: Aligning Textual Numerals and Visual Instances in Text-to-Video Diffusion Models

Abstract

Text-to-video diffusion models have enabled open-ended video synthesis, but often struggle with generating the correct number of objects specified in a prompt. We introduce NUMINA , a training-free identify-then-guide framework for improved numerical alignment. NUMINA identifies prompt-layout inconsistencies by selecting discriminative self- and cross-attention heads to derive a countable latent layout. It then refines this layout conservatively and modulates cross-attention to guide regeneration. On the introduced CountBench, NUMINA improves counting accuracy by up to 7.4% on Wan2.1-1.3B, and by 4.9% and 5.5% on 5B and 14B models, respectively. Furthermore, CLIP alignment is improved while maintaining temporal consistency. These results demonstrate that structural guidance complements seed search and prompt enhancement, offering a practical path toward count-accurate text-to-video diffusion. The code is available at https://github.com/H-EmbodVis/NUMINA.

Paper Structure

This paper contains 16 sections, 6 equations, 10 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Diffusion Transformer for Video Generation
  4. Video Editing for T2V Models
  5. Counting in Vision and Generation
  6. Preliminary
  7. Method
  8. Numerical Misalignment Identification
  9. Numerically Aligned Video Generation
  10. Experiments
  11. Experiment Setup
  12. Main Results
  13. Analysis and Ablation Study
  14. Conclusion
  15. Additional Results
  16. ...and 1 more sections

Figures (10)

  • Figure 1: We present NUMINA, a training-free framework that alleviates the misalignment between precise numerals and visual instances in text-to-video diffusion models. We significantly improve counting accuracy while maintaining natural layouts and temporal coherence.
  • Figure 2: Visualization of the cross-attention maps corresponding to different texts in the prompt. The highlighted areas represent a stronger level of attention between the pixels and the text.
  • Figure 3: The pipeline of our NUMINA follows a two-phase paradigm. Given a text prompt containing numerals, we first perform the numerical misalignment identification to extract explicitly countable layouts from attention maps. Based on the layout, we further conduct a refinement and a layout-guided generation for the numerically aligned video generation.
  • Figure 4: The PCA visualization of self-attention maps for Wan2.1-1.3B. (a) Different attention heads naturally capture diverse spatial patterns. (b) We select the head with the highest instance separability for countable layout construction.
  • Figure 5: Qualitative comparison of NUMINA with the most advanced commercial models.
  • ...and 5 more figures