Modeling Romanized Hindi and Bengali: Dataset Creation and Multilingual LLM Integration

TL;DR

The paper addresses the paucity of large-scale transliteration resources for Indo-Aryan languages written in Roman script by creating IndoTranslit, a dataset with over 2.7 million Hindi/Bengali transliteration pairs spanning clean, synthetic, and noisy variants. It trains a compact 60M-parameter MarianSeq2Seq model with a shared 32k subword vocabulary to transliterate Romanized inputs to native scripts in a multilingual setting, demonstrating strong BLEU and CER metrics surpassing existing baselines. Key contributions include the large-scale, mixed-quality dataset, a multilingual transliteration model, and open-source resources enabling reproducibility and downstream use. The work has practical impact for improving search, sentiment analysis, and conversational systems in South Asia, especially for users who encode native languages in Roman script and frequently switch languages. Future work aims to extend coverage to more Indo-Aryan languages and develop on-device transliteration for real-time applications.

Abstract

The development of robust transliteration techniques to enhance the effectiveness of transforming Romanized scripts into native scripts is crucial for Natural Language Processing tasks, including sentiment analysis, speech recognition, information retrieval, and intelligent personal assistants. Despite significant advancements, state-of-the-art multilingual models still face challenges in handling Romanized script, where the Roman alphabet is adopted to represent the phonetic structure of diverse languages. Within the South Asian context, where the use of Romanized script for Indo-Aryan languages is widespread across social media and digital communication platforms, such usage continues to pose significant challenges for cutting-edge multilingual models. While a limited number of transliteration datasets and models are available for Indo-Aryan languages, they generally lack sufficient diversity in pronunciation and spelling variations, adequate code-mixed data for large language model (LLM) training, and low-resource adaptation. To address this research gap, we introduce a novel transliteration dataset for two popular Indo-Aryan languages, Hindi and Bengali, which are ranked as the 3rd and 7th most spoken languages worldwide. Our dataset comprises nearly 1.8 million Hindi and 1 million Bengali transliteration pairs. In addition to that, we pre-train a custom multilingual seq2seq LLM based on Marian architecture using the developed dataset. Experimental results demonstrate significant improvements compared to existing relevant models in terms of BLEU and CER metrics.

Figures (4)

• Figure 1: Variations in Romanizing words for same sentence
• Figure 2: Distribution of character and word counts in Hindi and Bengali transliteration datasets.
• Figure 3: Vocabulary growth: cumulative number of distinct tokens observed over the dataset.
• Figure 4: Sentence length vs. average CER score for Hindi, Bengali, and Multi-Trans models