Evaluating Cross-Lingual Unlearning in Multilingual Language Models

TL;DR

It is demonstrated that multilingual forgetting depends on geometry in weight space, motivating subspace-based approaches for future unlearning systems.

Abstract

We present the first comprehensive evaluation of cross-lingual unlearning in multilingual LLMs. Using translated TOFU benchmarks in seven language/script variants, we test major unlearning algorithms and show that most fail to remove facts outside the training language, even when utility remains high. However, subspace-projection consistently outperforms the other methods, achieving strong cross-lingual forgetting with minimal degradation. Analysis of learned task subspaces reveals a shared interlingua structure: removing this shared subspace harms all languages, while removing language-specific components selectively affects one. These results demonstrate that multilingual forgetting depends on geometry in weight space, motivating subspace-based approaches for future unlearning systems.

Figures (11)

• Figure 1: Examples of question-answer pairs from all seven translations of the TOFU dataset.
• Figure 2: Comparison of unlearning performance on the English-English TOFU benchmark maini2024tofutaskfictitiousunlearning for three of our models: Llama 3, Mixtral, and Aya 23. See Table \ref{['tab:english_english']} for full results.
• Figure 3: Unlearning performance for non-multilingual models (Llama-2, Mistral) evaluated on English/Spanish TOFU dataset for all unlearning algorithms. See Table \ref{['tab:crosslingual_gen1']} for full results.
• Figure 4: Unlearning for multilingual models (Llama-3, Mixtral, Aya-23) evaluated on English/Spanish TOFU. See Table \ref{['tab:crosslingual_gen2']} for full results.
• Figure 5: UNLEARN for multilingual models (Llama-3, Mixtral, Aya-23) on all languages in TOFU. See Table \ref{['tab:crosslingual_all']} for full results.