Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multilingual Amnesia: On the Transferability of Unlearning in Multilingual LLMs

Alireza Dehghanpour Farashah, Aditi Khandelwal, Marylou Fauchard, Zhuan Shi, Negar Rostamzadeh, Golnoosh Farnadi

TL;DR

This study investigates multilingual unlearning in large language models by evaluating data- and concept-unlearning across ten languages using the Aya-Expanse-8B model. It extends two benchmarks, TOFU and SeeGULL, through translation to a diverse language set and employs three gradient-based unlearning objectives (GradDiff, GradDiff-KL, NPO) to balance forgetting with utility. The findings indicate that forgetting effects are largely language-specific with limited cross-lingual transfer, though partial transfer occurs between typologically related languages, and syntactic similarity emerges as the strongest predictor of transfer. The work highlights asymmetries in cross-lingual unlearning, reveals the impact of language resource levels on stability, and argues for language-aware unlearning strategies to ensure safe and fair deployment in multilingual settings.

Abstract

As multilingual large language models become more widely used, ensuring their safety and fairness across diverse linguistic contexts presents unique challenges. While existing research on machine unlearning has primarily focused on monolingual settings, typically English, multilingual environments introduce additional complexities due to cross-lingual knowledge transfer and biases embedded in both pretraining and fine-tuning data. In this work, we study multilingual unlearning using the Aya-Expanse 8B model under two settings: (1) data unlearning and (2) concept unlearning. We extend benchmarks for factual knowledge and stereotypes to ten languages through translation: English, French, Arabic, Japanese, Russian, Farsi, Korean, Hindi, Hebrew, and Indonesian. These languages span five language families and a wide range of resource levels. Our experiments show that unlearning in high-resource languages is generally more stable, with asymmetric transfer effects observed between typologically related languages. Furthermore, our analysis of linguistic distances indicates that syntactic similarity is the strongest predictor of cross-lingual unlearning behavior.

Multilingual Amnesia: On the Transferability of Unlearning in Multilingual LLMs

TL;DR

This study investigates multilingual unlearning in large language models by evaluating data- and concept-unlearning across ten languages using the Aya-Expanse-8B model. It extends two benchmarks, TOFU and SeeGULL, through translation to a diverse language set and employs three gradient-based unlearning objectives (GradDiff, GradDiff-KL, NPO) to balance forgetting with utility. The findings indicate that forgetting effects are largely language-specific with limited cross-lingual transfer, though partial transfer occurs between typologically related languages, and syntactic similarity emerges as the strongest predictor of transfer. The work highlights asymmetries in cross-lingual unlearning, reveals the impact of language resource levels on stability, and argues for language-aware unlearning strategies to ensure safe and fair deployment in multilingual settings.

Abstract

As multilingual large language models become more widely used, ensuring their safety and fairness across diverse linguistic contexts presents unique challenges. While existing research on machine unlearning has primarily focused on monolingual settings, typically English, multilingual environments introduce additional complexities due to cross-lingual knowledge transfer and biases embedded in both pretraining and fine-tuning data. In this work, we study multilingual unlearning using the Aya-Expanse 8B model under two settings: (1) data unlearning and (2) concept unlearning. We extend benchmarks for factual knowledge and stereotypes to ten languages through translation: English, French, Arabic, Japanese, Russian, Farsi, Korean, Hindi, Hebrew, and Indonesian. These languages span five language families and a wide range of resource levels. Our experiments show that unlearning in high-resource languages is generally more stable, with asymmetric transfer effects observed between typologically related languages. Furthermore, our analysis of linguistic distances indicates that syntactic similarity is the strongest predictor of cross-lingual unlearning behavior.
Paper Structure (22 sections, 5 equations, 14 figures, 6 tables)

This paper contains 22 sections, 5 equations, 14 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Machine Unlearning
  4. Multilingual LLMs
  5. Multilingual Unlearning
  6. Constructing Multilingual Unlearning Benchmarks
  7. Unlearning Objectives and Evaluation
  8. Data Unlearning
  9. Concept Unlearning
  10. Results and Analysis
  11. Data Unlearning: Localized Effects and Linguistic Correlations
  12. Concept Unlearning: Linguistic Asymmetry in Bias Mitigation
  13. Conclusion
  14. Limitations
  15. Acknowledgments
  16. ...and 7 more sections

Figures (14)

  • Figure 1: Framework for analyzing cross-lingual unlearning. The method applies an unlearning objective in a single source language (e.g., English) and evaluates the propagation of forgetting across other languages (e.g., French, Hindi) to measure transfer effects.
  • Figure 2: Cross-lingual Data Unlearning Efficacy: Heatmaps showing the ratio between the model’s probability on the forget set after unlearning and the corresponding probability under the finetuned baseline. Rows indicate the language in which unlearning is applied, while columns represent the language used for evaluation. Results are shown for three methods: GradDiff, GradDiff-KL, and NPO. Lower values correspond to stronger unlearning. Both axes are ordered according to the language resource level.
  • Figure 3: Cross-lingual Data Unlearning Retention: Heatmaps showing the ratio between the model’s probability on the retain set after unlearning and the corresponding probability under the finetuned baseline. Rows indicate the language in which unlearning is applied, while columns represent the language used for evaluation. Results are shown for three methods: GradDiff, GradDiff-KL, and NPO. Lower values indicate stronger side effects of unlearning on the retain set, while higher values reflect better retention. Both axes are ordered according to the language resource level.
  • Figure 4: Pairwise Syntactic Distances. Distances between the ten study languages derived from the URIEL typological database.
  • Figure 5: Comparison of model outputs after unlearning via GradDiff on English versus French for the same question on Aya model. The left panel shows the results for unlearning in English and the right panel shows the results for unlearning in French. This illustrates optional asymmetry in cross-lingual transfer, where unlearning in a relatively lower-resource language (French) may impact the high-resource language (English) more than the reverse.
  • ...and 9 more figures