Can Machine Translation Bridge Multilingual Pretraining and Cross-lingual Transfer Learning?

TL;DR

The paper investigates whether using a machine translation (MT) objective as continued pretraining can improve cross-lingual transfer in multilingual models. It compares multilingual LMs (e.g., mBART, mBERT, XLM-R) with MT variants on the XGLUE cross-lingual benchmarks, finding that MT-focused continual pretraining does not enhance cross-lingual performance and often degrades it. Through representational analyses using Centered Kernel Alignment (CKA) and weight-matrix investigations via singular value decomposition (SVD), the authors show that MT CP increases output separability and yields representations that are less aligned with other multilingual models, offering a possible explanation for the lack of transfer gains. The results challenge the assumption that explicit MT-driven cross-lingual alignment always benefits cross-lingual transfer and highlight the importance of representation geometry in multilingual learning and transfer scenarios.

Abstract

Multilingual pretraining and fine-tuning have remarkably succeeded in various natural language processing tasks. Transferring representations from one language to another is especially crucial for cross-lingual learning. One can expect machine translation objectives to be well suited to fostering such capabilities, as they involve the explicit alignment of semantically equivalent sentences from different languages. This paper investigates the potential benefits of employing machine translation as a continued training objective to enhance language representation learning, bridging multilingual pretraining and cross-lingual applications. We study this question through two lenses: a quantitative evaluation of the performance of existing models and an analysis of their latent representations. Our results show that, contrary to expectations, machine translation as the continued training fails to enhance cross-lingual representation learning in multiple cross-lingual natural language understanding tasks. We conclude that explicit sentence-level alignment in the cross-lingual scenario is detrimental to cross-lingual transfer pretraining, which has important implications for future cross-lingual transfer studies. We furthermore provide evidence through similarity measures and investigation of parameters that this lack of positive influence is due to output separability -- which we argue is of use for machine translation but detrimental elsewhere.

Figures (2)

• Figure 1: Representational similarity between mBART-based MT models and LMs
• Figure 2: Representational similarity between different languages with representations learned by LMs and MT models