Handwritten Text Recognition for Low Resource Languages

TL;DR

This work tackles paragraph-level handwritten text recognition for low-resource languages such as Hindi and Urdu by introducing BharatOCR, a segmentation-free architecture that processes entire paragraphs using a ViT-based image encoder, a Transformer decoder, and a pre-trained language model for refinement. The model leverages knowledge-distilled DeiT encoders, cross-modal and multi-modal attention, and a RoBERTa MLM to improve linguistic understanding, achieving strong, state-of-the-art results on Parimal Urdu/Hindi as well as public Urdu datasets. It also introduces the Parimal Urdu and Parimal Hindi datasets and demonstrates substantial performance gains over prior segmentation-based and transformer-based Urdu recognition methods, particularly at the character level. The approach reduces error propagation associated with segmentation and enables robust paragraph-level recognition across scripts, with potential extensions to other low-resource languages and scripts.

Abstract

Despite considerable progress in handwritten text recognition, paragraph-level handwritten text recognition, especially in low-resource languages, such as Hindi, Urdu and similar scripts, remains a challenging problem. These languages, often lacking comprehensive linguistic resources, require special attention to develop robust systems for accurate optical character recognition (OCR). This paper introduces BharatOCR, a novel segmentation-free paragraph-level handwritten Hindi and Urdu text recognition. We propose a ViT-Transformer Decoder-LM architecture for handwritten text recognition, where a Vision Transformer (ViT) extracts visual features, a Transformer decoder generates text sequences, and a pre-trained language model (LM) refines the output to improve accuracy, fluency, and coherence. Our model utilizes a Data-efficient Image Transformer (DeiT) model proposed for masked image modeling in this research work. In addition, we adopt a RoBERTa architecture optimized for masked language modeling (MLM) to enhance the linguistic comprehension and generative capabilities of the proposed model. The transformer decoder generates text sequences from visual embeddings. This model is designed to iteratively process a paragraph image line by line, called implicit line segmentation. The proposed model was evaluated using our custom dataset ('Parimal Urdu') and ('Parimal Hindi'), introduced in this research work, as well as two public datasets. The proposed model achieved benchmark results in the NUST-UHWR, PUCIT-OUHL, and Parimal-Urdu datasets, achieving character recognition rates of 96.24%, 92.05%, and 94.80%, respectively. The model also provided benchmark results using the Hindi dataset achieving a character recognition rate of 80.64%. The results obtained from our proposed model indicated that it outperformed several state-of-the-art Urdu text recognition methods.

Figures (6)

• Figure 1: The architecture consists of an image encoder, a transformer decoder, and a pre-trained language model. The image encoder extracts visual features from paragraph images, the transformer decoder generates an initial text sequence, and the language model refines it for coherence and accuracy.
• Figure 2: Our distillation procedure by including a new distillation token. It interacts with the class and patch tokens through the self-attention layers. This distillation token is employed similarly to the class token, except that on the network output, its objective is to reproduce the (hard) label predicted by the teacher instead of a true label. Both the class and distillation tokens input to the transformers are learned by back-propagation.
• Figure 3: RoBERTa—masked language modeling with the input sentence: The cat is eating some food
• Figure 4: Printed Urdu Text Paragraphs from different books of science and philosophy
• Figure 5: Handwritten text paragraphs from two authors with quite different writing styles from the Parimal Urdu dataset.