Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decipherment-Aware Multilingual Learning in Jointly Trained Language Models

Grandee Lee

TL;DR

This work reframes unsupervised cross-lingual learning in jointly trained multilingual models as a decipherment problem, linking cross-lingual transfer to the difficulty of aligning token distributions across languages. Through controlled decipherment experiments, it shows that data domain, lexical granularity, token order, and modeling objective jointly shape UCL performance, with MLM offering robust tolerance to misalignments. The study introduces an information-theoretic perspective, using metrics like $H(e|f)$ and $H(K|f)$ to explain upper bounds on cross-lingual signals and demonstrates that targeted lexical alignment via dictionaries can transfer decipherment gains to downstream cross-lingual tasks. Empirical results on downstream XTREME-style tasks confirm that alignment improves zero-shot transfer, though tokenization and word-type nuances modulate gains across classification, structured prediction, and retrieval tasks. The paper suggests future work on explicit mechanisms for many-to-many token alignment to further enhance multilingual transfer in real-world language pairs.

Abstract

The principle that governs unsupervised multilingual learning (UCL) in jointly trained language models (mBERT as a popular example) is still being debated. Many find it surprising that one can achieve UCL with multiple monolingual corpora. In this work, we anchor UCL in the context of language decipherment and show that the joint training methodology is a decipherment process pivotal for UCL. In a controlled setting, we investigate the effect of different decipherment settings on the multilingual learning performance and consolidate the existing opinions on the contributing factors to multilinguality. From an information-theoretic perspective we draw a limit to the UCL performance and demonstrate the importance of token alignment in challenging decipherment settings caused by differences in the data domain, language order and tokenization granularity. Lastly, we apply lexical alignment to mBERT and investigate the contribution of aligning different lexicon groups to downstream performance.

Decipherment-Aware Multilingual Learning in Jointly Trained Language Models

TL;DR

This work reframes unsupervised cross-lingual learning in jointly trained multilingual models as a decipherment problem, linking cross-lingual transfer to the difficulty of aligning token distributions across languages. Through controlled decipherment experiments, it shows that data domain, lexical granularity, token order, and modeling objective jointly shape UCL performance, with MLM offering robust tolerance to misalignments. The study introduces an information-theoretic perspective, using metrics like and to explain upper bounds on cross-lingual signals and demonstrates that targeted lexical alignment via dictionaries can transfer decipherment gains to downstream cross-lingual tasks. Empirical results on downstream XTREME-style tasks confirm that alignment improves zero-shot transfer, though tokenization and word-type nuances modulate gains across classification, structured prediction, and retrieval tasks. The paper suggests future work on explicit mechanisms for many-to-many token alignment to further enhance multilingual transfer in real-world language pairs.

Abstract

The principle that governs unsupervised multilingual learning (UCL) in jointly trained language models (mBERT as a popular example) is still being debated. Many find it surprising that one can achieve UCL with multiple monolingual corpora. In this work, we anchor UCL in the context of language decipherment and show that the joint training methodology is a decipherment process pivotal for UCL. In a controlled setting, we investigate the effect of different decipherment settings on the multilingual learning performance and consolidate the existing opinions on the contributing factors to multilinguality. From an information-theoretic perspective we draw a limit to the UCL performance and demonstrate the importance of token alignment in challenging decipherment settings caused by differences in the data domain, language order and tokenization granularity. Lastly, we apply lexical alignment to mBERT and investigate the contribution of aligning different lexicon groups to downstream performance.
Paper Structure (37 sections, 5 equations, 8 figures, 1 table)

This paper contains 37 sections, 5 equations, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. CLWE
  4. Jointly trained models
  5. Decipherment
  6. Deciperment Perspective on Joint Training
  7. Neural Decipherment
  8. Solving a substitution cipher
  9. Joint Training as Bidirectional Decipherment
  10. Decipherment Experiments
  11. Experiment Setup
  12. a) Effect of Domains
  13. b) Effect of Lexical Granularity
  14. c) Effect of Token Order
  15. d) Effect of Learning Objectives
  16. ...and 22 more sections

Figures (8)

  • Figure 1: The UCL performance of different objective functions under various decipherment settings.
  • Figure 2: The perplexity of the CLM and MLM model w.r.t. the number of permutations. Zero permutation corresponds to the original validation set, each permutation is a pair-wise swap of two randomly selected vocabulary indices.
  • Figure 3: Multilingual score w.r.t. JS divergence based on unigram distribution
  • Figure 4: The retrieval and BLI performance of a MLM model during training, under NT-NT setting.
  • Figure 5: The BLI performance of different objective functions under various decipherment scenarios.
  • ...and 3 more figures