TextInVision: Text and Prompt Complexity Driven Visual Text Generation Benchmark

TL;DR

Diffusion-based text-to-image models often fail to embed accurate visual text, limiting practical multimodal content generation. The paper introduces TextInVision, a large-scale benchmark that independently varies prompt complexity and text attributes and includes a dedicated VAE image set to probe text fidelity. Through OCR-based, CLIP-based, and human evaluations across seven state-of-the-art models and real-world prompts, the study shows that increasing prompt complexity and text length degrades text fidelity, while word frequency is only a weak predictor and the VAE acts as a major bottleneck. The benchmark identifies concrete failure points and provides a framework to diagnose and improve visual text rendering, with broad implications for education, advertising, and documentation.

Abstract

Generating images with embedded text is crucial for the automatic production of visual and multimodal documents, such as educational materials and advertisements. However, existing diffusion-based text-to-image models often struggle to accurately embed text within images, facing challenges in spelling accuracy, contextual relevance, and visual coherence. Evaluating the ability of such models to embed text within a generated image is complicated due to the lack of comprehensive benchmarks. In this work, we introduce TextInVision, a large-scale, text and prompt complexity driven benchmark designed to evaluate the ability of diffusion models to effectively integrate visual text into images. We crafted a diverse set of prompts and texts that consider various attributes and text characteristics. Additionally, we prepared an image dataset to test Variational Autoencoder (VAE) models across different character representations, highlighting that VAE architectures can also pose challenges in text generation within diffusion frameworks. Through extensive analysis of multiple models, we identify common errors and highlight issues such as spelling inaccuracies and contextual mismatches. By pinpointing the failure points across different prompts and texts, our research lays the foundation for future advancements in AI-generated multimodal content.

Figures (13)

• Figure 1: A comprehensive benchmark that enables separate analysis of prompt complexity and text attributes in evaluating T2I models. The benchmark features diverse prompt types—including Simple prompts (e.g., "a paper with the word 'text' written on it"), Complex prompts (e.g., "A sunny day at a theme park, a roller coaster with cars ascending the track, each car labeled with the text spelling out 'text', excited riders waving their hands"). It also incorporates a range of text attributes from single words to gibberish, numbers, special characters, phrases, and long texts.
• Figure 2: Generated images from multiple T2I models demonstrating common limitations in accuracy, clarity, and legibility.
• Figure 3: Average edit distance using real-world prompts.
• Figure 4: Average edit distance using different types of prompts and single words, compared across multiple models.
• Figure 5: Longer input texts tend to have a higher average edit distance and lead to lower image quality or legibility, further compounding the difficulty in accurately extracting the text.