Qalb: Largest State-of-the-Art Urdu Large Language Model for 230M Speakers with Systematic Continued Pre-training

TL;DR

Qalb addresses the underrepresentation of Urdu in large language models by starting from LLaMA-3.1 8B-Instruct and applying continued pre-training on a large, diverse Urdu–English corpus, followed by instruction fine-tuning on Alif Urdu-instruct. The approach yields state-of-the-art performance across seven Urdu benchmarks, achieving a weighted average of 90.34 and outperforming prior bests by substantial margins, including a 44.64-point gain over the base model. A 4-bit quantized version demonstrates strong performance with notable memory savings, suggesting practical deployment feasibility. The work validates that extensive language-specific pre-training, paired with targeted fine-tuning, is a scalable strategy for empowering low-resource languages and provides a reproducible methodology for similar languages.

Abstract

Despite remarkable progress in large language models, Urdu-a language spoken by over 230 million people-remains critically underrepresented in modern NLP systems. Existing multilingual models demonstrate poor performance on Urdu-specific tasks, struggling with the language's complex morphology, right-to-left Nastaliq script, and rich literary traditions. Even the base LLaMA-3.1 8B-Instruct model shows limited capability in generating fluent, contextually appropriate Urdu text. We introduce Qalb, an Urdu language model developed through a two-stage approach: continued pre-training followed by supervised fine-tuning. Starting from LLaMA 3.1 8B, we perform continued pre-training on a dataset of 1.97 billion tokens. This corpus comprises 1.84 billion tokens of diverse Urdu text-spanning news archives, classical and contemporary literature, government documents, and social media-combined with 140 million tokens of English Wikipedia data to prevent catastrophic forgetting. We then fine-tune the resulting model on the Alif Urdu-instruct dataset. Through extensive evaluation on Urdu-specific benchmarks, Qalb demonstrates substantial improvements, achieving a weighted average score of 90.34 and outperforming the previous state-of-the-art Alif-1.0-Instruct model (87.1) by 3.24 points, while also surpassing the base LLaMA-3.1 8B-Instruct model by 44.64 points. Qalb achieves state-of-the-art performance with comprehensive evaluation across seven diverse tasks including Classification, Sentiment Analysis, and Reasoning. Our results demonstrate that continued pre-training on diverse, high-quality language data, combined with targeted instruction fine-tuning, effectively adapts foundation models to low-resource languages.

Figures (3)

• Figure 1: The Qalb Urdu Language Model Development Pipeline. This flowchart visualizes the complete end-to-end process, from data collection and cleaning to continued pre-training, instruction fine-tuning, and final performance evaluation against benchmarks.
• Figure 2: Training and validation loss progression during continued pre-training on Urdu dataset over 7,500 steps. The blue line shows training loss decreasing from 1.07 to 0.77, while orange points indicate validation loss evaluated at regular intervals, closely tracking the training loss. The red line displays perplexity on the right y-axis, with the first measurement at step 2,500 showing a value of 2.35, indicating substantial Urdu language learning had already occurred in the initial training phase. Perplexity continues declining to approximately 2.20 by step 7,500, demonstrating the model's improving ability to predict Urdu text throughout training.
• Figure 3: Qualitative comparison of model outputs across representative tasks. The figure displays the original Urdu inputs and outputs from Qalb and Alif, highlighting differences in instruction adherence and generation quality.