Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PsOCR: Benchmarking Large Multimodal Models for Optical Character Recognition in Low-resource Pashto Language

Ijazul Haq, Yingjie Zhang, Irfan Ali Khan

TL;DR

PsOCR addresses the data scarcity barrier for Pashto OCR by introducing the first publicly available, large-scale synthetic dataset of one million images with word/line/document-level annotations across ~1,000 fonts. A 10K-PsOCR benchmark evaluates zero-shot performance of state-of-the-art large multimodal models, including open-source Llama, Florence, Qwen-3B/7B and proprietary GPT-4o, Gemini, Claude, Grok. Gemini achieves the best overall OCR performance (CER ~0.10, WER ~0.31), while Qwen-7B stands out among open-source models, indicating strong zero-shot capabilities and a solid baseline for future fine-tuning on Pashto and related Perso-Arabic scripts. The study highlights font-family diversity as a major challenge and provides a foundation for broader OCR research in low-resource scripts, with data availability on public platforms and mechanisms for access to the full training set.

Abstract

This paper evaluates the performance of Large Multimodal Models (LMMs) on Optical Character Recognition (OCR) in the low-resource Pashto language. Natural Language Processing (NLP) in Pashto faces several challenges due to the cursive nature of its script and a scarcity of structured datasets. To address this, we developed a synthetic Pashto OCR dataset, PsOCR, consisting of one million images annotated with bounding boxes at word, line, and document levels, suitable for training and evaluating models based on different architectures, including Convolutional Neural Networks (CNNs) and Transformers. PsOCR covers variations across 1,000 unique font families, colors, image sizes, and layouts. A benchmark subset of 10K images was selected to evaluate the performance of several LMMs, including seven open-source models: DeepSeek's Janus, InternVL, MiniCPM, Florence, and Qwen (3B and 7B), and four closed-source models: GPT-4o, Gemini, Claude, and Grok. Experimental results demonstrate that Gemini achieves the best performance among all models, whereas among open-source models, Qwen-7B stands out. This work provides an insightful assessment of the capabilities and limitations of current LMMs for OCR tasks in Pashto and establishes a foundation for further research not only in Pashto OCR but also for other similar scripts such as Arabic, Persian, and Urdu. PsOCR is available at https://github.com/zirak-ai/PashtoOCR.

PsOCR: Benchmarking Large Multimodal Models for Optical Character Recognition in Low-resource Pashto Language

TL;DR

PsOCR addresses the data scarcity barrier for Pashto OCR by introducing the first publicly available, large-scale synthetic dataset of one million images with word/line/document-level annotations across ~1,000 fonts. A 10K-PsOCR benchmark evaluates zero-shot performance of state-of-the-art large multimodal models, including open-source Llama, Florence, Qwen-3B/7B and proprietary GPT-4o, Gemini, Claude, Grok. Gemini achieves the best overall OCR performance (CER ~0.10, WER ~0.31), while Qwen-7B stands out among open-source models, indicating strong zero-shot capabilities and a solid baseline for future fine-tuning on Pashto and related Perso-Arabic scripts. The study highlights font-family diversity as a major challenge and provides a foundation for broader OCR research in low-resource scripts, with data availability on public platforms and mechanisms for access to the full training set.

Abstract

This paper evaluates the performance of Large Multimodal Models (LMMs) on Optical Character Recognition (OCR) in the low-resource Pashto language. Natural Language Processing (NLP) in Pashto faces several challenges due to the cursive nature of its script and a scarcity of structured datasets. To address this, we developed a synthetic Pashto OCR dataset, PsOCR, consisting of one million images annotated with bounding boxes at word, line, and document levels, suitable for training and evaluating models based on different architectures, including Convolutional Neural Networks (CNNs) and Transformers. PsOCR covers variations across 1,000 unique font families, colors, image sizes, and layouts. A benchmark subset of 10K images was selected to evaluate the performance of several LMMs, including seven open-source models: DeepSeek's Janus, InternVL, MiniCPM, Florence, and Qwen (3B and 7B), and four closed-source models: GPT-4o, Gemini, Claude, and Grok. Experimental results demonstrate that Gemini achieves the best performance among all models, whereas among open-source models, Qwen-7B stands out. This work provides an insightful assessment of the capabilities and limitations of current LMMs for OCR tasks in Pashto and establishes a foundation for further research not only in Pashto OCR but also for other similar scripts such as Arabic, Persian, and Urdu. PsOCR is available at https://github.com/zirak-ai/PashtoOCR.
Paper Structure (33 sections, 5 equations, 12 figures, 6 tables)

This paper contains 33 sections, 5 equations, 12 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Pashto OCR
  4. Synthetic OCR Datasets
  5. OCR Benchmarks for LMMs
  6. Dataset Development
  7. Text Corpus Collection
  8. Text Cleaning and Preprocessing
  9. Text to Image Conversion
  10. Dataset Composition
  11. Granularity
  12. Font Variation
  13. Images Size and Aspect Ratio
  14. Themes and Colors
  15. Other Variations
  16. ...and 18 more sections

Figures (12)

  • Figure 1: Sample images from the dataset showing different levels of granularity and bounding box annotation
  • Figure 2: A sample image from the dataset in PNG format, with the corresponding details and annotation information in JSON format
  • Figure 3: A few examples of font families in PsOCR dataset
  • Figure 4: Colors and themes in the dataset and their ratio
  • Figure 5: Dataset Statistics
  • ...and 7 more figures