OCRTurk: A Comprehensive OCR Benchmark for Turkish

TL;DR

OCRTurk addresses the shortage of Turkish document-parsing benchmarks by introducing a diverse, real-world benchmark of $180$ Turkish pages across four document categories and three difficulty levels, with element-level evaluation for raw texts, tables, equations, and figures. The framework defines dedicated metrics for each element type and evaluates seven OCR models, finding PaddleOCR to have the strongest overall performance, while variations emerge across document types and element types (e.g., DeepSeekOCR leading in figure extraction and NanonetsOCR2 excelling in equations). The authors provide a full data-generation and annotation pipeline, publicly release datasets and scripts, and offer insights into how Turkish morphology and layout complexity impact OCR robustness. This benchmark aims to drive progress in Turkish OCR and document understanding by enabling fair, multi-faceted evaluation and the potential for an online leaderboard and future dataset expansion.

Abstract

Document parsing is now widely used in applications, such as large-scale document digitization, retrieval-augmented generation, and domain-specific pipelines in healthcare and education. Benchmarking these models is crucial for assessing their reliability and practical robustness. Existing benchmarks mostly target high-resource languages and provide limited coverage for low-resource settings, such as Turkish. Moreover, existing studies on Turkish document parsing lack a standardized benchmark that reflects real-world scenarios and document diversity. To address this gap, we introduce OCRTurk, a Turkish document parsing benchmark covering multiple layout elements and document categories at three difficulty levels. OCRTurk consists of 180 Turkish documents drawn from academic articles, theses, slide decks, and non-academic articles. We evaluate seven OCR models on OCRTurk using element-wise metrics. Across difficulty levels, PaddleOCR achieves the strongest overall results, leading most element-wise metrics except figures and attaining high Normalized Edit Distance scores in easy, medium, and hard subsets. We also observe performance variation by document type. Models perform well on non-academic documents, while slideshows become the most challenging.

Figures (4)

• Figure 1: Comparison of the models DeepSeekOCR, Docling, PaddleOCR, HuanyanOCR, NanonetsOCR2, Nvidia Nemotron v1.1, and OlmOCR2 under easy, medium, and hard data. The scores are the averages of the NED metric scores for raw texts, tables, and equations of the data within the same difficulty. The NED scores are subtracted from 1 ($\text{Score} = 1 - \text{NED}$) for better comparison. The average score of the models within each difficulty level is given as the dashed red line.
• Figure 2: Comparison of the models DeepSeekOCR, Docling, PaddleOCR, HuanyanOCR, NanonetsOCR2, Nvidia Nemotron v1.1, and OlmOCR2 under the categories academic documents, non-academic documents, theses, and slideshows. The scores are the averages of the NED metric scores for raw texts, tables, and equations of the data within the same difficulty. The NED scores are subtracted from 1 ($\text{Score} = 1 - \text{NED}$) for better comparison. The average score of the models within each category is given as the dashed red line.
• Figure 3: Comparison of the tables between the ground truth (a) and the model output (b). As the structure of the table in the model output is incorrect, it spans to more than 50 lines. Thus, we ignore the rest of the model output in this figure.
• Figure 4: A figure of the model output which shows the misinterpretation of the table as an image, which should have been five separate images in a table, mixed with texts.