Language Detection for Transliterated Content

TL;DR

The paper tackles language identification for content transliterated into the English alphabet, using Hindi and Russian as case studies. It adopts a BERT-based classifier trained on a synthetic romanized dataset generated by translating native scripts via Google Translate API and transliterating with Polyglot, achieving a validation accuracy of $99\%$. Key contributions include a practical data-generation pipeline, an end-to-end transliteration-then-translation workflow, and demonstrated robustness of BERT for romanized language ID. The work enables scalable processing of multilingual online content, with potential impact on content moderation, analytics, and cross-language communication.

Abstract

In the contemporary digital era, the Internet functions as an unparalleled catalyst, dismantling geographical and linguistic barriers particularly evident in texting. This evolution facilitates global communication, transcending physical distances and fostering dynamic cultural exchange. A notable trend is the widespread use of transliteration, where the English alphabet is employed to convey messages in native languages, posing a unique challenge for language technology in accurately detecting the source language. This paper addresses this challenge through a dataset of phone text messages in Hindi and Russian transliterated into English utilizing BERT for language classification and Google Translate API for transliteration conversion. The research pioneers innovative approaches to identify and convert transliterated text, navigating challenges in the diverse linguistic landscape of digital communication. Emphasizing the pivotal role of comprehensive datasets for training Large Language Models LLMs like BERT, our model showcases exceptional proficiency in accurately identifying and classifying languages from transliterated text. With a validation accuracy of 99% our models robust performance underscores its reliability. The comprehensive exploration of transliteration dynamics supported by innovative approaches and cutting edge technologies like BERT, positions our research at the forefront of addressing unique challenges in the linguistic landscape of digital communication. Beyond contributing to language identification and transliteration capabilities this work holds promise for applications in content moderation, analytics and fostering a globally connected community engaged in meaningful dialogue.

Figures (6)

• Figure 1: Model Architecture
• Figure 2: Dataset Generation
• Figure 3: Model Parameters
• Figure 4: Confusion Matrix
• Figure 5: Evaluation Metrics