Deciphering the Underserved: Benchmarking LLM OCR for Low-Resource Scripts

TL;DR

The paper investigates the feasibility of using GPT-4o for OCR on low-resource scripts (Urdu, Albanian, Tajik) with English as a benchmark. It develops a 2,520-image multilingual dataset with controlled word counts, font sizes, backgrounds, and blur to stress OCR under real-world conditions, and evaluates zero-shot GPT-4o OCR using CER, WER, and BLEU. Findings show pronounced difficulty for Urdu and Tajik under longer texts, smaller fonts, low contrast, and blur, while Albanian and English remain robust, underscoring the need for annotated data and script-aware fine-tuning. The work offers a publicly available benchmark and highlights the practical impact of inclusive OCR, advocating for broader language coverage, dataset expansion, and cost-effective alternatives to advance digitization for underserved languages.

Abstract

This study investigates the potential of Large Language Models (LLMs), particularly GPT-4o, for Optical Character Recognition (OCR) in low-resource scripts such as Urdu, Albanian, and Tajik, with English serving as a benchmark. Using a meticulously curated dataset of 2,520 images incorporating controlled variations in text length, font size, background color, and blur, the research simulates diverse real-world challenges. Results emphasize the limitations of zero-shot LLM-based OCR, particularly for linguistically complex scripts, highlighting the need for annotated datasets and fine-tuned models. This work underscores the urgency of addressing accessibility gaps in text digitization, paving the way for inclusive and robust OCR solutions for underserved languages.

Figures (21)

• Figure 1: WER with Increasing Word Count
• Figure 2: CER with Increasing Word Count
• Figure 3: BLEU score with Increasing Word Count
• Figure 4: WER with Increasing Font Size
• Figure 5: CER with Increasing Font Size