Exploring the Role of Transliteration in In-Context Learning for Low-resource Languages Written in Non-Latin Scripts

TL;DR

The paper investigates whether transliteration can boost in-context learning for decoder-only LLMs when handling low-resource languages written in non-Latin scripts. It introduces three transliteration-based prompts—orig, Latin-script transliteration, and a combined prompt—and evaluates them across six models on NER, SIB200, and Taxi1500 tasks. Key findings show transliteration yields strong gains for sequential labeling and mixed, task-dependent effects for classification, with model type and size significantly influencing outcomes. The work highlights transliteration as a promising tool for cross-script transfer in ICL but acknowledges limitations such as moderate model sizes and dataset scope, suggesting directions for scaling to larger models and broader task sets. Overall, the study provides practical guidance on when and how transliteration can enhance ICL for non-Latin-script languages in decoder-only LLMs, informing future research and prompt-design practices.

Abstract

Decoder-only large language models (LLMs) excel in high-resource languages across various tasks through few-shot or even zero-shot in-context learning (ICL). However, their performance often does not transfer well to low-resource languages, especially those written in non-Latin scripts. Inspired by recent work that leverages transliteration in encoder-only models, we investigate whether transliteration is also effective in improving LLMs' performance for low-resource languages written in non-Latin scripts. To this end, we propose three prompt templates, where the target-language text is represented in (1) its original script, (2) Latin script, or (3) both. We apply these methods to several representative LLMs of different sizes on various tasks including text classification and sequential labeling. Our findings show that the effectiveness of transliteration varies by task type and model size. For instance, all models benefit from transliterations for sequential labeling (with increases of up to 25%).

Figures (4)

• Figure 1: Results of LLaMA7B, Mistral7B, BLOOM7B and BLOOM3B on NER task. By leveraging transliteration, Script$_{\{\text{Latn}\}}$ or Script$_{\{\text{Combined}\}}$ consistently improve the performance on NER across all models.
• Figure 2: Illustration of our framework. We use Uromanhermjakob-etal-2018-box to transliterate non-Latin texts (sentence-level for text classification, and word-level for sequential labeling). We propose three prompts: Script$_{\{\text{Orig}\}}$ (the original text is used), Script$_{\{\text{Latn}\}}$ (the Latin-script transliteration is used), and Script$_{\{\text{Combined}\}}$ (transliteration is used as an augmentation to the original text).
• Figure 3: Performance on NER task averaged by languages of the same script. Transliterations are generally effective in improving the ICL across all models and scripts: Script$_{\{\text{Latn}\}}$ or Script$_{\{\text{Combined}\}}$ outperforms Script$_{\{\text{Orig}\}}$.
• Figure 4: Three types of prompt templates (Script$_{\{\text{Orig}\}}$, Script$_{\{\text{Latn}\}}$ and Script$_{\{\text{Combined}\}}$) that are used for each task. We follow the prompt templates in lin2024mala500 for the Script$_{\{\text{Orig}\}}$, where the target-langauge text is represented in the original script. We use Latin-script transliterations obtained by Uromanhermjakob-etal-2018-box for Script$_{\{\text{Latn}\}}$. Script$_{\{\text{Combined}\}}$ leverages both the original script and its Latin transliteration.