How Transliterations Improve Crosslingual Alignment

TL;DR

This paper investigates why transliteration-augmented pretraining improves crosslingual alignment without parallel data. By defining four sentence-level similarities and conducting case studies on Polish-Ukrainian and Hindi-Urdu, the authors show that simply adding transliterations increases overall similarity but not alignment; auxiliary objectives, especially Transliteration Contrastive Modeling (TCM), improve the ability to distinguish matched from random pairs, yielding better alignment. However, stronger crosslingual alignment does not consistently translate to improved downstream zero-shot transfer, indicating a nuanced relationship between alignment and task performance. The findings suggest transliteration-based signals are robust and beneficial for alignment, but further work is needed to connect alignment improvements with practical transfer gains, particularly for token-level alignment and complex downstream tasks.

Abstract

Recent studies have shown that post-aligning multilingual pretrained language models (mPLMs) using alignment objectives on both original and transliterated data can improve crosslingual alignment. This improvement further leads to better crosslingual transfer performance. However, it remains unclear how and why a better crosslingual alignment is achieved, as this technique only involves transliterations, and does not use any parallel data. This paper attempts to explicitly evaluate the crosslingual alignment and identify the key elements in transliteration-based approaches that contribute to better performance. For this, we train multiple models under varying setups for two pairs of related languages: (1) Polish and Ukrainian and (2) Hindi and Urdu. To assess alignment, we define four types of similarities based on sentence representations. Our experimental results show that adding transliterations alone improves the overall similarities, even for random sentence pairs. With the help of auxiliary transliteration-based alignment objectives, especially the contrastive objective, the model learns to distinguish matched from random pairs, leading to better crosslingual alignment. However, we also show that better alignment does not always yield better downstream performance, suggesting that further research is needed to clarify the connection between alignment and performance. The code implementation is based on \url{https://github.com/cisnlp/Transliteration-PPA}.

Figures (10)

• Figure 1: Histograms of similarities for matched sentence pairs and random pairs. Adding transliterated data in pretraining improves the overall similarities for both matched and random pairs (Model-2). Leveraging auxiliary objectives improves the model's ability to differentiate between matched and random sentence pairs (Model-3,-4,-5).
• Figure 2: Comparison of different types of similarities. We observe that the inclusion of the transliterated data not only improve those similarities that involve transliterations (i.e., $\text{sim}(\mathcal{M}(s), \mathcal{M}(r_s))$, $\text{sim}(\mathcal{M}(r_s), \mathcal{M}(t))$ and $\text{sim}(\mathcal{M}(r_s), \mathcal{M}(r_t))$), but also the similarity between the translation pairs, i.e., $\text{sim}(\mathcal{M}(s), \mathcal{M}(t))$.
• Figure 3: Dynamics of four types of similarities during training progression (from 2K to 50K checkpoints). We calculate the average of all paired sentences in SR-B for each type of similarity in each checkpoint.
• Figure 4: Histograms of similarities for matched sentence pairs and random pairs. Adding transliterated data in pretraining improves the overall similarities for both matched and random pairs. Leveraging auxiliary objectives improves the model's ability to differentiate between matched sentence pairs from random sentence pairs.
• Figure 5: Comparison of different types of similarities (measured using SR-F).