Early evidence of how LLMs outperform traditional systems on OCR/HTR tasks for historical records

TL;DR

This study tests whether large language models (GPT-4o and Claude Sonnet 3.5) can directly transcribe historical handwritten tabular records, comparing them to traditional OCR/HTR systems. By evaluating line-by-line and whole-scan inputs with CER and BLEU, plus human judgments, the authors show that LLMs generally outperform conventional OCR/HTR pipelines, with two-shot prompting delivering the best alignment to ground truth in respective input modes. The work also reveals that BLEU and CER can diverge and that header content disproportionately influences scores, suggesting BLEU as a useful metric for longer, structured transcriptions. The results have practical implications for fast, scalable digitization of historical documents and guide future research on model ensembles, digits-focused fine-tuning, and benchmark development.

Abstract

We explore the ability of two LLMs -- GPT-4o and Claude Sonnet 3.5 -- to transcribe historical handwritten documents in a tabular format and compare their performance to traditional OCR/HTR systems: EasyOCR, Keras, Pytesseract, and TrOCR. Considering the tabular form of the data, two types of experiments are executed: one where the images are split line by line and the other where the entire scan is used as input. Based on CER and BLEU, we demonstrate that LLMs outperform the conventional OCR/HTR methods. Moreover, we also compare the evaluated CER and BLEU scores to human evaluations to better judge the outputs of whole-scan experiments and understand influential factors for CER and BLEU. Combining judgments from all the evaluation metrics, we conclude that two-shot GPT-4o for line-by-line images and two-shot Claude Sonnet 3.5 for whole-scan images yield the transcriptions of the historical records most similar to the ground truth.

Figures (7)

• Figure 1: BLEU score comparisons for each method on the line-by-line dataset: GPT 4-o, Claude Sonnet 3.5, OCR tools and Fine-tuned TrOCR. A higher BLEU means a higher similarity with the GT. For each method, the mean score is in a dashed line, and the median is in a bold line. The maximum order in the n-gram used to calculate BLEU is 3. The texts are stripped, lowered and unidecoded before calculating the scores.
• Figure 2: CER score comparisons for each method on the line-by-line dataset: GPT 4-o, Claude Sonnet 3.5, OCR tools and Fine-tuned TrOCR. The Y-axis is zoomed at [-0.5, 2]. A lower CER means a higher similarity with the GT. The maximum CER value observed is 64. For each method, the mean score is in a dashed line, and the median is in a bold line. The texts are pre-processed as in Fig. \ref{['fig:bleu_perline']}.
• Figure 3: BLEU score comparisons of each method between the whole scan experiments (), and the line-by-line experiments (). A higher BLEU means a higher similarity with the GT. For each method, the mean score is in a dashed line, and the median is in a bold line. The maximum order in the n-gram used to calculate BLEU is 3 for the line-by-line experiments and 4 for the whole scan experiments. TrOCR50 is not performed with the whole scan dataset.
• Figure 4: CER score comparisons of each method between the whole scan experiments (), and the line-by-line experiments (). A lower CER means a higher similarity with the GT. The maximum CER value observed is 1.21 for the whole scan and 64 for the line-by-line experiments. The Y-axis is zoomed at [-0.5, 2]. For each method, the mean score is in a dashed line, and the median is in a bold line. TrOCR50 is not performed with the whole scan dataset.
• Figure 5: BLEU and CER scores with the headers () and without the headers () for one-example and two-example prompts by GPT-4o and Claude Sonnet 3.5. After removing the headers, Claude Two-Example has the highest BLEU and lowest CER on average.