Multimodal LLMs for OCR, OCR Post-Correction, and Named Entity Recognition in Historical Documents

TL;DR

The paper demonstrates that multimodal large language models can substantially improve transcription, post-correction, and information extraction from historical documents, specifically German city directories (1754–1870). By comparing GPT-4o and Gemini 2.0 Flash against traditional OCR systems, the study shows that multimodal OCR post-correction yields state-of-the-art accuracy (normalized CER near 0.8–1.3%), and that NER with structured parsing is feasible even from noisy transcriptions or directly from images. The authors introduce iterative prompting to maximize mLLM performance and release code and data to support replication and benchmarking. Overall, the work provides early evidence that mLLMs could drastically accelerate historical data collection and dataset construction, potentially reshaping archival transcription practices.

Abstract

We explore how multimodal Large Language Models (mLLMs) can help researchers transcribe historical documents, extract relevant historical information, and construct datasets from historical sources. Specifically, we investigate the capabilities of mLLMs in performing (1) Optical Character Recognition (OCR), (2) OCR Post-Correction, and (3) Named Entity Recognition (NER) tasks on a set of city directories published in German between 1754 and 1870. First, we benchmark the off-the-shelf transcription accuracy of both mLLMs and conventional OCR models. We find that the best-performing mLLM model significantly outperforms conventional state-of-the-art OCR models and other frontier mLLMs. Second, we are the first to introduce multimodal post-correction of OCR output using mLLMs. We find that this novel approach leads to a drastic improvement in transcription accuracy and consistently produces highly accurate transcriptions (<1% CER), without any image pre-processing or model fine-tuning. Third, we demonstrate that mLLMs can efficiently recognize entities in transcriptions of historical documents and parse them into structured dataset formats. Our findings provide early evidence for the long-term potential of mLLMs to introduce a paradigm shift in the approaches to historical data collection and document transcription.

Figures (9)

• Figure 1: The First Page of each Directory in our Dataset
• Figure 2: Multimodal LLMs for OCR and OCR Post-Correction
• Figure 3: Multimodal LLMs for NER
• Figure 4: Prompt for PNG-to-CSV NER task
• Figure 5: Visualization of NER Results for Gemini 2.0 Flash