UrduLM: A Resource-Efficient Monolingual Urdu Language Model

TL;DR

UrduLM tackles the lack of Urdu monolingual LLMs by building a 100M-parameter decoder-only model trained on a curated 33GB Urdu corpus. It introduces a script-aware 32k BPE tokenizer that reduces tokenization overhead and demonstrates competitive few-shot and LLM-as-a-Judge performance versus much larger multilingual models, achieving 66.6% sentiment accuracy and 30.59 BLEU on grammar correction. The work provides a complete, openly released pipeline (data, tokenizer, model weights, benchmarks) that serves as a baseline and scalable framework for other low-resource languages. It discusses design trade-offs and establishes generalizable principles for corpus size, tokenizer design, and model scale, arguing for monolingual pretraining when data and resources permit. The results suggest practical impact for efficient, culturally aware NLP in Urdu and similar underrepresented languages.

Abstract

Urdu, spoken by 230 million people worldwide, lacks dedicated transformer-based language models and curated corpora. While multilingual models provide limited Urdu support, they suffer from poor performance, high computational costs, and cultural inaccuracies due to insufficient training data. To address these challenges, we present UrduLM, a pretrained Urdu monolingual language model trained in low-resource settings. We curate a 33GB Urdu corpus from diverse sources, develop a custom BPE tokenizer that reduces tokenization overhead by atleast 20-30% compared to multilingual alternatives, and pretrain a 100M-parameter decoder-only model. In few-shot evaluations, UrduLM achieves competitive performance with multilingual models up to 30x its size, reaching 66.6% accuracy on sentiment classification and BLEU scores exceeding 30 on grammar correction tasks. The complete methodology -- including corpus, tokenizer, model weights, and evaluation benchmarks -- is released openly to establish a baseline for Urdu NLP research and provide a scalable framework for other underrepresented languages.

Figures (7)

• Figure 1: Tokenizer efficiency comparison: Our custom tokenizers produce 20--30% fewer tokens than GPT-4 o200k_base across all vocabulary sizes.
• Figure 2: Overview of the data collection and preprocessing pipeline for the UrduLM pre-training dataset.
• Figure 3: Distribution of data sources in the UrduLM pretraining dataset.
• Figure 4: LLM-as-a-Judge evaluation: UrduLM-100M competitive despite 10--30$\times$ size disadvantage. Scores averaged over 50 diverse Urdu prompts across coherence, relevance, and fluency.
• Figure 5: Training and validation loss for UrduLM-100M with 10k vocabulary. Validation loss (orange) closely tracks training loss (blue), indicating healthy generalization.