T2VTextBench: A Human Evaluation Benchmark for Textual Control in Video Generation Models

TL;DR

T2VTextBench introduces a human-evaluated benchmark for on-screen text fidelity and temporal coherence in text-to-video models. The study evaluates ten modern models using a 73-prompt, six-category suite and a 0–1 human scoring scheme, revealing pervasive difficulties in rendering legible, consistent text and large instability across prompts and architectures. Ablation analyses on text transformations and randomness show only modest gains, with geometric changes being particularly challenging, while a pricing analysis identifies Wan 2.1 as cost-effective relative to quality. These findings underscore the need for explicit text modeling and enhanced temporal control in video synthesis to support practical applications in advertising, education, and beyond.

Abstract

Thanks to recent advancements in scalable deep architectures and large-scale pretraining, text-to-video generation has achieved unprecedented capabilities in producing high-fidelity, instruction-following content across a wide range of styles, enabling applications in advertising, entertainment, and education. However, these models' ability to render precise on-screen text, such as captions or mathematical formulas, remains largely untested, posing significant challenges for applications requiring exact textual accuracy. In this work, we introduce T2VTextBench, the first human-evaluation benchmark dedicated to evaluating on-screen text fidelity and temporal consistency in text-to-video models. Our suite of prompts integrates complex text strings with dynamic scene changes, testing each model's ability to maintain detailed instructions across frames. We evaluate ten state-of-the-art systems, ranging from open-source solutions to commercial offerings, and find that most struggle to generate legible, consistent text. These results highlight a critical gap in current video generators and provide a clear direction for future research aimed at enhancing textual manipulation in video synthesis.

Figures (16)

• Figure 1: T2V Model Generate Text Result Across Different Measures.
• Figure 2: T2V Model Generate Text Result Across Different Measures.
• Figure 3: Ablation Study on Text Generation under Different Transformations.
• Figure 4: Case Study on Text Transformations.
• Figure 5: Ablation Study of Prompts with Different Levels of Text Randomness.