Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What Causes Knowledge Loss in Multilingual Language Models?

Maria Khelli, Samuel Cahyawijaya, Ayu Purwarianti, Genta Indra Winata

TL;DR

This work investigates catastrophic forgetting in cross-lingual transfer for multilingual NLP by evaluating 52 languages with LoRA adapters under different parameter-sharing regimes. Through a lifelong-learning setup and a suite of metrics including CFT, CBT, MFT, and MBT, it reveals that non-Latin-script languages are more prone to forgetting, while Latin-script languages transfer more effectively. The study finds that partial sharing (NON-SHARED LoRA) often strikes a favorable balance between retaining prior knowledge and achieving strong cross-lingual performance, whereas fully shared adapters can incur large forgetting costs. These insights inform robust multilingual system design and suggest vitality-aware scheduling to improve continual learning in real-world multilingual settings.

Abstract

Cross-lingual transfer in natural language processing (NLP) models enhances multilingual performance by leveraging shared linguistic knowledge. However, traditional methods that process all data simultaneously often fail to mimic real-world scenarios, leading to challenges like catastrophic forgetting, where fine-tuning on new tasks degrades performance on previously learned ones. Our study explores this issue in multilingual contexts, focusing on linguistic differences affecting representational learning rather than just model parameters. We experiment with 52 languages using LoRA adapters of varying ranks to evaluate non-shared, partially shared, and fully shared parameters. Our aim is to see if parameter sharing through adapters can mitigate forgetting while preserving prior knowledge. We find that languages using non-Latin scripts are more susceptible to catastrophic forgetting, whereas those written in Latin script facilitate more effective cross-lingual transfer.

What Causes Knowledge Loss in Multilingual Language Models?

TL;DR

This work investigates catastrophic forgetting in cross-lingual transfer for multilingual NLP by evaluating 52 languages with LoRA adapters under different parameter-sharing regimes. Through a lifelong-learning setup and a suite of metrics including CFT, CBT, MFT, and MBT, it reveals that non-Latin-script languages are more prone to forgetting, while Latin-script languages transfer more effectively. The study finds that partial sharing (NON-SHARED LoRA) often strikes a favorable balance between retaining prior knowledge and achieving strong cross-lingual performance, whereas fully shared adapters can incur large forgetting costs. These insights inform robust multilingual system design and suggest vitality-aware scheduling to improve continual learning in real-world multilingual settings.

Abstract

Cross-lingual transfer in natural language processing (NLP) models enhances multilingual performance by leveraging shared linguistic knowledge. However, traditional methods that process all data simultaneously often fail to mimic real-world scenarios, leading to challenges like catastrophic forgetting, where fine-tuning on new tasks degrades performance on previously learned ones. Our study explores this issue in multilingual contexts, focusing on linguistic differences affecting representational learning rather than just model parameters. We experiment with 52 languages using LoRA adapters of varying ranks to evaluate non-shared, partially shared, and fully shared parameters. Our aim is to see if parameter sharing through adapters can mitigate forgetting while preserving prior knowledge. We find that languages using non-Latin scripts are more susceptible to catastrophic forgetting, whereas those written in Latin script facilitate more effective cross-lingual transfer.
Paper Structure (34 sections, 13 equations, 7 figures, 3 tables)

This paper contains 34 sections, 13 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Task Setup
  4. Training Methods
  5. MULTI.
  6. MONO.
  7. VANILLA.
  8. SHARED LoRA.
  9. NON-SHARED LoRA.
  10. Model Parameters Adaptation
  11. Experimental Setup
  12. Datasets
  13. Hyper-parameters
  14. Evaluation Metrics
  15. Performance Shift
  16. ...and 19 more sections

Figures (7)

  • Figure 1: Pipeline for various approaches in lifelong learning. In our lifelong learning framework, we employ a LoRA-based approach where the parameters of the base model, denoted as $\theta$, remain fixed, and for VANILLA, the model parameters are updated at all times. We explore the phenomenon of multilingual knowledge loss by comparing the effects of training with both shared and non-shared parameters.
  • Figure 2: Performance results after training each language over the time.
  • Figure 3: Performance change after training a certain language on x-axis in sequential training (VANILLA). Chinese (zh-CN) exhibits the most significant performance decline, while German (de-DE) serves as the most effective donor language, enhancing overall performance.
  • Figure 4: Comparison results between XLM-R and E5 models.
  • Figure 5: Heatmap of Multi-hop Backward Transfer (MBT), illustrates how training on later languages affects earlier ones over increasing hop distances (rows 0–9). Cooler colors indicate positive backward transfer, while warmer colors reflect degradation in performance. Notable negative effects are observed in languages like ja-JP, zh-CN, and zh-TW.
  • ...and 2 more figures