Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates

Atsuki Yamaguchi, Terufumi Morishita, Aline Villavicencio, Nikolaos Aletras

TL;DR

This paper tackles catastrophic forgetting when adapting instruct LLMs to target languages under low-resource conditions by introducing Source-Shielded Updates (SSU). SSU uses a source-data-driven parameter importance scoring method (e.g., Wanda) to identify critical weights, applies a column-wise freezing mask before adaptation, and performs continual pre-training on unlabeled target data with these masks. Experiments across five languages and two model scales show that SSU preserves core source abilities—such as instruction-following and safety—while achieving target-language performance competitive with full fine-tuning, often surpassing it on 7B models. The work also provides extensive analyses of freezing ratios, alternative scoring methods, calibration data effects, and qualitative behavior, and discusses practical implications and reproducibility. Overall, SSU offers a scalable, principled approach to expanding the linguistic reach of instruct LLMs without costly labeled data or substantial forgetting.

Abstract

Expanding the linguistic diversity of instruct large language models (LLMs) is crucial for global accessibility but is often hindered by the reliance on costly specialized target language labeled data and catastrophic forgetting during adaptation. We tackle this challenge under a realistic, low-resource constraint: adapting instruct LLMs using only unlabeled target language data. We introduce Source-Shielded Updates (SSU), a selective parameter update strategy that proactively preserves source knowledge. Using a small set of source data and a parameter importance scoring method, SSU identifies parameters critical to maintaining source abilities. It then applies a column-wise freezing strategy to protect these parameters before adaptation. Experiments across five typologically diverse languages and 7B and 13B models demonstrate that SSU successfully mitigates catastrophic forgetting. It reduces performance degradation on monolingual source tasks to just 3.4% (7B) and 2.8% (13B) on average, a stark contrast to the 20.3% and 22.3% from full fine-tuning. SSU also achieves target-language performance highly competitive with full fine-tuning, outperforming it on all benchmarks for 7B models and the majority for 13B models.

Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates

TL;DR

This paper tackles catastrophic forgetting when adapting instruct LLMs to target languages under low-resource conditions by introducing Source-Shielded Updates (SSU). SSU uses a source-data-driven parameter importance scoring method (e.g., Wanda) to identify critical weights, applies a column-wise freezing mask before adaptation, and performs continual pre-training on unlabeled target data with these masks. Experiments across five languages and two model scales show that SSU preserves core source abilities—such as instruction-following and safety—while achieving target-language performance competitive with full fine-tuning, often surpassing it on 7B models. The work also provides extensive analyses of freezing ratios, alternative scoring methods, calibration data effects, and qualitative behavior, and discusses practical implications and reproducibility. Overall, SSU offers a scalable, principled approach to expanding the linguistic reach of instruct LLMs without costly labeled data or substantial forgetting.

Abstract

Expanding the linguistic diversity of instruct large language models (LLMs) is crucial for global accessibility but is often hindered by the reliance on costly specialized target language labeled data and catastrophic forgetting during adaptation. We tackle this challenge under a realistic, low-resource constraint: adapting instruct LLMs using only unlabeled target language data. We introduce Source-Shielded Updates (SSU), a selective parameter update strategy that proactively preserves source knowledge. Using a small set of source data and a parameter importance scoring method, SSU identifies parameters critical to maintaining source abilities. It then applies a column-wise freezing strategy to protect these parameters before adaptation. Experiments across five typologically diverse languages and 7B and 13B models demonstrate that SSU successfully mitigates catastrophic forgetting. It reduces performance degradation on monolingual source tasks to just 3.4% (7B) and 2.8% (13B) on average, a stark contrast to the 20.3% and 22.3% from full fine-tuning. SSU also achieves target-language performance highly competitive with full fine-tuning, outperforming it on all benchmarks for 7B models and the majority for 13B models.

Paper Structure

This paper contains 50 sections, 2 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Language Adaptation.
  4. Catastrophic Forgetting.
  5. Selective Parameter Updates.
  6. SSU: A Source-Focused Selective Parameter Update Approach.
  7. SSU: Selective Parameter Updates via Importance Freezing
  8. Source-Driven Parameter Importance Scoring
  9. Column-wise Masking
  10. Continual Pre-training
  11. Experimental Setup
  12. Source Models
  13. Target Languages
  14. Calibration and Training Data
  15. Baselines
  16. ...and 35 more sections

Figures (2)

  • Figure 1: Overview of Source-Shielded Update (SSU). The method comprises three stages: importance scoring, column-wise mask generation, and continual pre-training on unlabeled target language data with the masks.
  • Figure 2: Model performance (SSU-Wanda, HFT, GMT) on Igbo as target language across freezing ratios. The dashed red line indicates Source performance (omitted for MT and SUM due to very low scores). Some data points for HFT and GMT are also omitted due to extremely low performance.