TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models

TL;DR

TransliCo tackles the script barrier in multilingual pretrained language models by jointly fine-tuning with MLM on original and Latin-transliterated data and a Transliteration Contrastive Modeling objective to align cross-script representations. By constructing Latin transliterations of a portion of the pretraining corpus and contrasting pairs (original, transliteration) against negatives, it encourages script-neutral embeddings and improves zero-shot crosslingual transfer. Empirical results on Glot500-m (yielding Furina) show robust gains across sentence retrieval, classification, and sequence labeling tasks, with notable improvements in the Indic group; analysis and an ablation study confirm the necessity of both MLM and TCM and the benefit of incorporating Latin-script data. The framework is lightweight, scalable, and publicly released, offering a practical path to enhance cross-script transfer for low-resource languages without transliterating user queries at inference time.

Abstract

The world's more than 7000 languages are written in at least 293 scripts. Due to various reasons, many closely related languages use different scripts, which poses a difficulty for multilingual pretrained language models (mPLMs) in learning crosslingual knowledge through lexical overlap. As a consequence, mPLMs are faced with a script barrier: representations from different scripts are located in different subspaces, which can result in crosslingual transfer involving languages of different scripts performing suboptimally. To address this problem, we propose TransliCo, a framework that optimizes the Transliteration Contrastive Modeling (TCM) objective to fine-tune an mPLM by contrasting sentences in its training data and their transliterations in a unified script (in our case Latin), which enhances uniformity in the representation space for different scripts. Using Glot500-m, an mPLM pretrained on over 500 languages, as our source model, we fine-tune it on a small portion (5%) of its training data, and refer to the resulting model as Furina. We show that Furina not only better aligns representations from distinct scripts but also outperforms the original Glot500-m on various zero-shot crosslingual transfer tasks. Additionally, we achieve consistent improvement in a case study on the Indic group where the languages exhibit areal features but use different scripts. We make our code and models publicly available.

Figures (5)

• Figure 1: An illustration of applying TransliCo to a single batch of data during fine-tuning. The training data is used by the two training objectives in TransliCo: Masked Language Modeling (MLM) and Transliteration Contrastive Modeling (TCM). MLM is applied to both the original sentences and their Latin transliterations. TCM is used to learn better-aligned cross-script representations by contrasting the positive pairs (paired data connected with red lines) against the negative pairs (the remaining samples connected with blue lines).
• Figure 2: Overview of TransliCo. We perform Masked Language Modeling for a sentence in its original script and its transliteration in the Latin script. Meanwhile, we calculate the sequence representations of the paired input by mean pooling their 8th layer output (ignoring the special token except for [mask] token). We then perform Transliteration Contrastive Modeling on the paired representations against negative pairs (not shown) in a batch.
• Figure 3: (i) PCA of sentence representations from layer 8 (mean-pooling of contextualized token embeddings, dim=768) of Furina (Subfigure 1) and Glot500-m (Subfigure 2). Points are sentence representations. Colors indicate scripts. (ii) Pairwise cosine similarity between centroids of scripts of Furina (Subfigure 3) and Glot500-m (Subfigure 4). Furina better represents scripts in several cases, e.g., it better aligns related scripts Latn and Cyrl, and, it better separates the unrelated scripts Cyrl and Mlym compared to Glot500-m.
• Figure 4: Visualizations of sentence representations from all layers (mean-pooling the contextualized token embeddings) of Furina. The original dimension is 768 and we use PCA to select the first two principal components. Each point corresponds to a sentence. Different colors indicate distinct scripts.
• Figure 5: Visualizations of sentence representations from all layers (mean-pooling the contextualized token embeddings) of Glot500-m. The original dimension is 768 and we use PCA to select the first two principal components. Points indicate sentence representations and different colors indicate distinct scripts.