Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

One Script Instead of Hundreds? On Pretraining Romanized Encoder Language Models

Benedikt Ebing, Lennart Keller, Goran Glavaš

TL;DR

This study asks whether pretraining encoder LMs on romanized text harms high-resource languages and whether romanization can safely enhance cross-lingual sharing. By training monolingual and multilingual models from scratch on romanized vs. native-script data across six languages and two romanization schemes, the authors quantify script-specific information loss and cross-lingual interference. They find minimal performance loss for segmental scripts, with higher-fidelity romanization partially mitigating losses for Chinese and Japanese; crucially, increased subword overlap does not induce negative interference in multilingual settings. The work also shows romanization improves tokenizer fertility, enabling more efficient encoding with little to no accuracy cost, suggesting romanized pretraining as a practical path for scalable multilingual models across diverse scripts.

Abstract

Exposing latent lexical overlap, script romanization has emerged as an effective strategy for improving cross-lingual transfer (XLT) in multilingual language models (mLMs). Most prior work, however, focused on setups that favor romanization the most: (1) transfer from high-resource Latin-script to low-resource non-Latin-script languages and/or (2) between genealogically closely related languages with different scripts. It thus remains unclear whether romanization is a good representation choice for pretraining general-purpose mLMs, or, more precisely, if information loss associated with romanization harms performance for high-resource languages. We address this gap by pretraining encoder LMs from scratch on both romanized and original texts for six typologically diverse high-resource languages, investigating two potential sources of degradation: (i) loss of script-specific information and (ii) negative cross-lingual interference from increased vocabulary overlap. Using two romanizers with different fidelity profiles, we observe negligible performance loss for languages with segmental scripts, whereas languages with morphosyllabic scripts (Chinese and Japanese) suffer degradation that higher-fidelity romanization mitigates but cannot fully recover. Importantly, comparing monolingual LMs with their mLM counterpart, we find no evidence that increased subword overlap induces negative interference. We further show that romanization improves encoding efficiency (i.e., fertility) for segmental scripts at a negligible performance cost.

One Script Instead of Hundreds? On Pretraining Romanized Encoder Language Models

TL;DR

This study asks whether pretraining encoder LMs on romanized text harms high-resource languages and whether romanization can safely enhance cross-lingual sharing. By training monolingual and multilingual models from scratch on romanized vs. native-script data across six languages and two romanization schemes, the authors quantify script-specific information loss and cross-lingual interference. They find minimal performance loss for segmental scripts, with higher-fidelity romanization partially mitigating losses for Chinese and Japanese; crucially, increased subword overlap does not induce negative interference in multilingual settings. The work also shows romanization improves tokenizer fertility, enabling more efficient encoding with little to no accuracy cost, suggesting romanized pretraining as a practical path for scalable multilingual models across diverse scripts.

Abstract

Exposing latent lexical overlap, script romanization has emerged as an effective strategy for improving cross-lingual transfer (XLT) in multilingual language models (mLMs). Most prior work, however, focused on setups that favor romanization the most: (1) transfer from high-resource Latin-script to low-resource non-Latin-script languages and/or (2) between genealogically closely related languages with different scripts. It thus remains unclear whether romanization is a good representation choice for pretraining general-purpose mLMs, or, more precisely, if information loss associated with romanization harms performance for high-resource languages. We address this gap by pretraining encoder LMs from scratch on both romanized and original texts for six typologically diverse high-resource languages, investigating two potential sources of degradation: (i) loss of script-specific information and (ii) negative cross-lingual interference from increased vocabulary overlap. Using two romanizers with different fidelity profiles, we observe negligible performance loss for languages with segmental scripts, whereas languages with morphosyllabic scripts (Chinese and Japanese) suffer degradation that higher-fidelity romanization mitigates but cannot fully recover. Importantly, comparing monolingual LMs with their mLM counterpart, we find no evidence that increased subword overlap induces negative interference. We further show that romanization improves encoding efficiency (i.e., fertility) for segmental scripts at a negligible performance cost.
Paper Structure (13 sections, 3 equations, 4 figures, 19 tables)

This paper contains 13 sections, 3 equations, 4 figures, 19 tables.

Table of Contents

  1. Motivation and Background
  2. Romanization Approaches
  3. Experiments
  4. Results and Analyses
  5. Conclusion
  6. Limitations
  7. Acknowledgements
  8. Pretraining
  9. Downstream Fine-Tuning
  10. Details on Fertility
  11. Details on Loss of information
  12. Detailed Results: Main Results and Impact of Tokenization
  13. Detailed Results: Fertility vs. Information Loss

Figures (4)

  • Figure 1: Fertility vs. token merging for monolingual models with URoman: (a) relative change in fertility (lower is better) for MonoUroman compared to MonoNat; (b) relative change in vocabulary size (higher is better) of MonoNat when romanizing its vocabulary with URoman (i.e., subwords conflated due to same romanization).
  • Figure 2: Absolute performance difference (avg. across all six languages and five tasks) between MonoNat and MonoUroman for different vocabulary sizes.
  • Figure 3: Fertility vs. Token Collapse for monolingual models using UConv: (a) relative change in fertility (lower is better) for MonoUconv compared to MonoNat; (b) relative change in the vocabulary size (higher is better) of MonoNat when romanizing each subword in its vocabulary using UConv (i.e., subwords get merged due to the same romanization).
  • Figure 4: Fertility vs. token merging for multilingual models: (a) relative change in fertility (lower is better) for MonoUroman compared to MonoNat; (b) relative change in the vocabulary size (higher is better) of MonoNat when romanizing each subword in its vocabulary (i.e., subwords get merged due to the same romanization).