Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VDE Bench: Evaluating The Capability of Image Editing Models to Modify Visual Documents

Hongzhu Yi, Yujia Yang, Yuanxiang Wang, Zhenyu Guan, Jiahuan Chen, Chenxi Bao, Tiankun Yang, Yixuan Yuan, Tianyu Zong, Xinming Wang, Tao Yu, Ruiwen Tao, Haijin Liang, Jin Ma, Jinwen Luo, Yeshani Xinyu Zuo, Jungang Xu

TL;DR

This work targets visual document image editing by foregrounding multilingual, densely textual documents and recognizing the inadequacy of English-only benchmarks. It introduces VDE Bench, a rigorously annotated benchmark with English and Chinese data and an OCR-parsing level evaluation framework that decouples spatial and textual assessment. The evaluation combines an OCR-based pipeline (PaddleOCR-VL) with spatial (IoU) and textual (CDM, BLEU-4, TEDS-like) metrics to benchmark both global and region-level editing performance across multiple models. Findings reveal trade-offs between background preservation and text modification accuracy, especially for Chinese text and text-addition tasks, and demonstrate strong alignment between automated metrics and human judgments, establishing VDE Bench as a reliable tool for advancing multilingual visual document editing.

Abstract

In recent years, multimodal image editing models have achieved substantial progress, enabling users to manipulate visual content through natural language in a flexible and interactive manner. Nevertheless, an important yet insufficiently explored research direction remains visual document image editing, which involves modifying textual content within images while faithfully preserving the original text style and background context. Existing approaches, including AnyText, GlyphControl, and TextCtrl, predominantly focus on English-language scenarios and documents with relatively sparse textual layouts, thereby failing to adequately address dense, structurally complex documents or non-Latin scripts such as Chinese. To bridge this gap, we propose \textbf{V}isual \textbf{D}oc \textbf{E}dit Bench(VDE Bench), a rigorously human-annotated and evaluated benchmark specifically designed to assess image editing models on multilingual and complex visual document editing tasks. The benchmark comprises a high-quality dataset encompassing densely textual documents in both English and Chinese, including academic papers, posters, presentation slides, examination materials, and newspapers. Furthermore, we introduce a decoupled evaluation framework that systematically quantifies editing performance at the OCR parsing level, enabling fine-grained assessment of text modification accuracy. Based on this benchmark, we conduct a comprehensive evaluation of representative state-of-the-art image editing models. Manual verification demonstrates a strong consistency between human judgments and automated evaluation metrics. VDE Bench constitutes the first systematic benchmark for evaluating image editing models on multilingual and densely textual visual documents.

VDE Bench: Evaluating The Capability of Image Editing Models to Modify Visual Documents

TL;DR

This work targets visual document image editing by foregrounding multilingual, densely textual documents and recognizing the inadequacy of English-only benchmarks. It introduces VDE Bench, a rigorously annotated benchmark with English and Chinese data and an OCR-parsing level evaluation framework that decouples spatial and textual assessment. The evaluation combines an OCR-based pipeline (PaddleOCR-VL) with spatial (IoU) and textual (CDM, BLEU-4, TEDS-like) metrics to benchmark both global and region-level editing performance across multiple models. Findings reveal trade-offs between background preservation and text modification accuracy, especially for Chinese text and text-addition tasks, and demonstrate strong alignment between automated metrics and human judgments, establishing VDE Bench as a reliable tool for advancing multilingual visual document editing.

Abstract

In recent years, multimodal image editing models have achieved substantial progress, enabling users to manipulate visual content through natural language in a flexible and interactive manner. Nevertheless, an important yet insufficiently explored research direction remains visual document image editing, which involves modifying textual content within images while faithfully preserving the original text style and background context. Existing approaches, including AnyText, GlyphControl, and TextCtrl, predominantly focus on English-language scenarios and documents with relatively sparse textual layouts, thereby failing to adequately address dense, structurally complex documents or non-Latin scripts such as Chinese. To bridge this gap, we propose \textbf{V}isual \textbf{D}oc \textbf{E}dit Bench(VDE Bench), a rigorously human-annotated and evaluated benchmark specifically designed to assess image editing models on multilingual and complex visual document editing tasks. The benchmark comprises a high-quality dataset encompassing densely textual documents in both English and Chinese, including academic papers, posters, presentation slides, examination materials, and newspapers. Furthermore, we introduce a decoupled evaluation framework that systematically quantifies editing performance at the OCR parsing level, enabling fine-grained assessment of text modification accuracy. Based on this benchmark, we conduct a comprehensive evaluation of representative state-of-the-art image editing models. Manual verification demonstrates a strong consistency between human judgments and automated evaluation metrics. VDE Bench constitutes the first systematic benchmark for evaluating image editing models on multilingual and densely textual visual documents.
Paper Structure (30 sections, 6 equations, 5 figures, 4 tables)

This paper contains 30 sections, 6 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image Editing Models.
  4. Image Editing Benchmarks.
  5. Visual Text Generation and Modification Benchmarks.
  6. VDE Bench
  7. Data Collection Stage
  8. Image Generation Stage
  9. Manual Review Stage
  10. Human Annotation Rules Alignment:
  11. Text Modification Annotation:
  12. Statistical Analysis
  13. Evaluation Method and Metrics
  14. Spatial Matching Algorithm
  15. Matching Strategy
  16. ...and 15 more sections

Figures (5)

  • Figure 1: Overview of the VDE Bench pipeline, including document data sourcing, instruction generation and image editing, followed by rigorous manual review and OCR-based global and local evaluation.
  • Figure 2: An example data sample from VDE Bench.
  • Figure 3: Overview of the data distribution.
  • Figure 4: Overview of the evaluation pipeline. The model-generated images are cropped according to the edited region boxes provided in the groud truth data to obtain the local regions. OCR recognition is then performed on both the global and local regions using PaddleOCR-VL, and the discrepancies between the OCR results and the groud truth are subsequently calculated.
  • Figure 5: Correlation between human rankings and automated rankings.The horizontal axis represents the human ranking results, and the vertical axis represents the automated ranking results.