TextEditBench: Evaluating Reasoning-aware Text Editing Beyond Rendering

TL;DR

TextEditBench tackles the challenge of editing text embedded in images by enforcing reasoning about semantic, contextual, and cross-modal constraints. It introduces a dual-track evaluation—pixel-level fidelity and MLLM-based semantic assessment—and a Semantic Expectation metric with an explicit Knowledge Prompt. The dataset comprises 1,196 annotated instances across 14 domains, 6 task types, and 12 sub-tasks with diverse layouts and multilingual content. Experiments show current editors handle simple edits but falter on multi-step reasoning and layout-constrained semantic edits, underscoring the need for reasoning-aware benchmarks.

Abstract

Text rendering has recently emerged as one of the most challenging frontiers in visual generation, drawing significant attention from large-scale diffusion and multimodal models. However, text editing within images remains largely unexplored, as it requires generating legible characters while preserving semantic, geometric, and contextual coherence. To fill this gap, we introduce TextEditBench, a comprehensive evaluation benchmark that explicitly focuses on text-centric regions in images. Beyond basic pixel manipulations, our benchmark emphasizes reasoning-intensive editing scenarios that require models to understand physical plausibility, linguistic meaning, and cross-modal dependencies. We further propose a novel evaluation dimension, Semantic Expectation (SE), which measures reasoning ability of model to maintain semantic consistency, contextual coherence, and cross-modal alignment during text editing. Extensive experiments on state-of-the-art editing systems reveal that while current models can follow simple textual instructions, they still struggle with context-dependent reasoning, physical consistency, and layout-aware integration. By focusing evaluation on this long-overlooked yet fundamental capability, TextEditBench establishes a new testing ground for advancing text-guided image editing and reasoning in multimodal generation.

Figures (27)

• Figure 1: Overview of TextEditBench .TextEditBench covers diverse text-in-image editing types such as translation, replacement, color and rotation adjustment, text removal, scaling, and creation, spanning both visual fidelity and reasoning-intensive semantic edits. For clarity, we highlight the major regions of modification with red rectangle.
• Figure 2: Data collection and annotation pipeline of TextEditBench . The dataset is constructed from a balanced mixture of manually produced instances with paired input/ground-truth images and web-sourced instances with input images only. For each sample, a human annotator first designs the editing instruction and attribute labels; a customized GPT5 prompt then refines and normalizes the text; finally, a human reviewer validates the outputs before inclusion in the benchmark.
• Figure 3: Data distribution.TextEditBench includes synthetic and real-world images, covering common text-in-image cases.
• Figure 4: Overview of the evaluation pipeline. The evaluation framework combines both Semantic Metrics and Pixel-Level Fidelity Metrics. Left: GPT-4o evaluates text-guided edits across five complementary dimensions. When the edit involves higher-level semantic reasoning, an auxiliary knowledge prompt is optionally provided for SE assessment. Right: Pixel-level objective metrics are proposed to quantify preservation of unedited content.
• Figure 5: Illustration of the five dimensions of the Semantic Expectation (SE) metric. Key conceptual aspects are highlighted for clarity.