Conditions for Catastrophic Forgetting in Multilingual Translation

TL;DR

The work tackles catastrophic forgetting in multilingual MT during fine-tuning. By systematically varying base-model architecture, size, data volume, and fine-tuning method, the authors identify the relative model-data scale as a key determinant of forgetting and show that instruction-following ability, not architecture, largely governs retention. They find that parameter-efficient fine-tuning (LoRA) offers no clear advantage over full fine-tuning for mitigating forgetting in their setups, while cross-lingual alignment can promote positive transfer for unseen pairs but struggles with unseen targets. The results yield actionable guidance for deploying multilingual MT systems: choose larger base models when heavy fine-tuning is needed, leverage instruction-following prompts and in-language instructions, and consider cross-lingual alignment primarily to aid unseen-pair transfer; limitations include English-centric data and computational constraints that limit exploration of even larger scales.

Abstract

Fine-tuning multilingual foundation models on specific languages often induces catastrophic forgetting, degrading performance on languages unseen in fine-tuning. While this phenomenon is widely-documented, the literature presents fragmented results about when forgetting occurs. To address this ambiguity, we conduct a systematic empirical study using machine translation as a testbed to identify the conditions that trigger catastrophic forgetting in multilingual fine-tuning. Through controlled experiments across different model architectures, data scales, and fine-tuning approaches, we reveal that the relative scale between model and data size is a primary determinant of forgetting. Moreover, we demonstrate that a model's instruction-following ability is more critical for retaining multilingual knowledge than its architecture. Contrary to assumptions, parameter-efficient fine-tuning offers no clear advantage over full fine-tuning in mitigating forgetting. Lastly, we show that cross-lingual alignment can mitigate forgetting while also facilitating positive transfer to unseen target languages.

Figures (6)

• Figure 1: Selectively fine-tuning on some languages or translation directions may lead to positive transfer (left) or catastrophic forgetting (right).
• Figure 2: Gain-forgetting plots on the small dataset (117K sentence pairs). Catastrophic forgetting is minimal, except unseen language pairs on Llama (addressed later in \ref{['tab:chap6_inlang_instruct']}).
• Figure 3: Gain-forgetting plots on the large dataset (54M sentence pairs). Catastrophic forgetting is more severe, especially with translation-specific models where they show performance collapse approaching 0 BLEU.
• Figure 4: Controlled analysis of data volume effects by subsampled portions of the large dataset. Forgetting becomes severe as fine-tuning data amount increases.
• Figure 5: Cross-lingual alignment effects on translation-specific models on the large dataset. Alignment methods bring gains in unseen language pairs, but suffer from persistent forgetting in unseen target languages.