MEMLA: Enhancing Multilingual Knowledge Editing with Neuron-Masked Low-Rank Adaptation

TL;DR

This work tackles multilingual knowledge editing for large language models by introducing the MKEB benchmark (12 languages) and a neuron-masked LoRA method, MEMLA, that identifies language-specific versus language-independent knowledge neurons to propagate updates efficiently across languages. MEMLA combines integrated gradients-based neuron identification with two editors (Language-Specific and Language-Independent) and neuron masks to achieve precise edits with minimal disruption to unrelated knowledge, significantly improving cross-lingual reliability, generality, locality, and multi-hop reasoning. Empirical results show MEMLA surpasses strong baselines (e.g., MEMIT, MEND) in cross-lingual transfer and multi-hop tasks while maintaining downstream performance; ablations confirm the necessity of masks and both editors. The dataset and code release will enable broader evaluation and advancement of multilingual knowledge editing techniques in real-world, multilingual deployments.

Abstract

Knowledge editing aims to adjust the knowledge within large language models (LLMs) to prevent their responses from becoming obsolete or inaccurate. However, existing works on knowledge editing are primarily conducted in a single language, which is inadequate for multilingual language models. In this paper, we focus on multilingual knowledge editing (MKE), which requires propagating updates across multiple languages. This necessity poses a significant challenge for the task. Furthermore, the limited availability of a comprehensive dataset for MKE exacerbates this challenge, hindering progress in this area. Hence, we introduce the Multilingual Knowledge Editing Benchmark (MKEB), a novel dataset comprising 12 languages and providing a complete evaluation framework. Additionally, we propose a method that enhances Multilingual knowledge Editing with neuron-Masked Low-Rank Adaptation (MEMLA). Specifically, we identify two categories of knowledge neurons to improve editing precision. Moreover, we perform LoRA-based editing with neuron masks to efficiently modify parameters and facilitate the propagation of updates across multiple languages. Experiments demonstrate that our method outperforms existing baselines and significantly enhances the multi-hop reasoning capability of the edited model, with minimal impact on its downstream task performance. The dataset and code will be made publicly available.

Figures (5)

• Figure 1: An example of MKE: when a fact is updated in one language (e.g., English), the new fact is transferred to other languages (e.g., Chinese).
• Figure 2: The construction process of our MKEB dataset, which involves retrieving numerous facts from Wikidata, using handcrafted prompts to induce ChatGPT to generate data samples, and further processing these samples through filtering and translation.
• Figure 3: Distributions of top-10 relations in our dataset.
• Figure 4: The overall framework of MEMLA, where $W_{fc}^{\left(l\right)}$ and $W_{proj}^{\left(l\right)}$ denote the first and second weights of the MLP in the $l$-th layer, respectively. LIKNs represent language-independent knowledge neurons. LIE and LSE represent the language-independent editor and language-specific editor, respectively. LIM and LSM denote the language-independent neuron mask and the language-specific neuron mask, respectively.
• Figure 5: Performance of the edited model on downstream tasks.