OCR-Quality: A Human-Annotated Dataset for OCR Quality Assessment

TL;DR

OCR-Quality addresses the need for reliable OCR quality assessment in diverse real-world documents by providing a large, human-annotated dataset of 1,000 pages with four-level quality scores. The authors process pages using a state-of-the-art Vision-Language Model and release rich metadata, representative cases, and an evaluation protocol to support training and benchmarking of OCR verification and uncertainty estimation methods. Baseline results reveal that current VLM-based quality assessment remains challenging, with methods like Consensus Entropy offering particular strength for intermediate-quality outputs. Overall, OCR-Quality enables researchers and practitioners to develop robust quality gates, model selection criteria, and active-learning strategies for trustworthy OCR pipelines.

Abstract

We present OCR-Quality, a comprehensive human-annotated dataset designed for evaluating and developing OCR quality assessment methods. The dataset consists of 1,000 PDF pages converted to PNG images at 300 DPI, sampled from diverse real-world scenarios, including academic papers, textbooks, e-books, and multilingual documents. Each document has been processed using state-of-the-art Vision-Language Models (VLMs) and manually annotated with quality scores using a 4-level scoring system (1: Excellent, 2: Good, 3: Fair, 4: Poor). The dataset includes detailed source information, annotation guidelines, and representative cases across various difficulty levels. OCR-Quality addresses the critical need for reliable OCR quality assessment in real-world applications and provides a valuable benchmark for training and evaluating OCR verification systems. The dataset is publicly available at https://huggingface.co/datasets/Aslan-mingye/OCR-Quality .