Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Low-Resource Transliteration for Roman-Urdu and Urdu Using Transformer-Based Models

Umer Butt, Stalin Veranasi, Günter Neumann

TL;DR

This work proposes a transformer-based approach using the m2m100 multilingual translation model, enhanced with masked language modeling (MLM) pretraining and fine-tuning on both Roman-Urdu-Parl and the domain-diverse Dakshina dataset, and achieves strong transliteration performance.

Abstract

As the Information Retrieval (IR) field increasingly recognizes the importance of inclusivity, addressing the needs of low-resource languages remains a significant challenge. Transliteration between Urdu and its Romanized form, Roman Urdu, remains underexplored despite the widespread use of both scripts in South Asia. Prior work using RNNs on the Roman-Urdu-Parl dataset showed promising results but suffered from poor domain adaptability and limited evaluation. We propose a transformer-based approach using the m2m100 multilingual translation model, enhanced with masked language modeling (MLM) pretraining and fine-tuning on both Roman-Urdu-Parl and the domain-diverse Dakshina dataset. To address previous evaluation flaws, we introduce rigorous dataset splits and assess performance using BLEU, character-level BLEU, and CHRF. Our model achieves strong transliteration performance, with Char-BLEU scores of 96.37 for Urdu->Roman-Urdu and 97.44 for Roman-Urdu->Urdu. These results outperform both RNN baselines and GPT-4o Mini and demonstrate the effectiveness of multilingual transfer learning for low-resource transliteration tasks.

Low-Resource Transliteration for Roman-Urdu and Urdu Using Transformer-Based Models

TL;DR

This work proposes a transformer-based approach using the m2m100 multilingual translation model, enhanced with masked language modeling (MLM) pretraining and fine-tuning on both Roman-Urdu-Parl and the domain-diverse Dakshina dataset, and achieves strong transliteration performance.

Abstract

As the Information Retrieval (IR) field increasingly recognizes the importance of inclusivity, addressing the needs of low-resource languages remains a significant challenge. Transliteration between Urdu and its Romanized form, Roman Urdu, remains underexplored despite the widespread use of both scripts in South Asia. Prior work using RNNs on the Roman-Urdu-Parl dataset showed promising results but suffered from poor domain adaptability and limited evaluation. We propose a transformer-based approach using the m2m100 multilingual translation model, enhanced with masked language modeling (MLM) pretraining and fine-tuning on both Roman-Urdu-Parl and the domain-diverse Dakshina dataset. To address previous evaluation flaws, we introduce rigorous dataset splits and assess performance using BLEU, character-level BLEU, and CHRF. Our model achieves strong transliteration performance, with Char-BLEU scores of 96.37 for Urdu->Roman-Urdu and 97.44 for Roman-Urdu->Urdu. These results outperform both RNN baselines and GPT-4o Mini and demonstrate the effectiveness of multilingual transfer learning for low-resource transliteration tasks.

Paper Structure

This paper contains 17 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background on Romanization & Roman Urdu
  4. Methodology
  5. Correcting the Roman-Urdu-Parl Dataset Split
  6. Dataset Splitting Strategy
  7. Ensuring Data Integrity and Generalization
  8. Model Architecture and Tokenization
  9. Masked Language Modeling (MLM) Pretraining
  10. Pretraining Data and Training Setup
  11. Fine-Tuning for Transliteration
  12. Training Configuration
  13. Evaluation Metrics Selection
  14. Results and Discussion
  15. Conclusion and Future Work
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Examples of Romanization in English → Urdu and German → English. In both cases, the romanized sentences closely resemble the phonetic pronunciation in the native language.