Multilingual Knowledge Editing with Language-Agnostic Factual Neurons

TL;DR

This work introduces Language-Agnostic Factual Neurons (LAFNs) as FFN neurons that encode the same facts across languages, uncovering cross-language semantic connections. It then proposes LU-LAFNs, a two-stage method that (1) locates LAFNs using paraphrase-augmented multilingual prompts and (2) updates their values to achieve synchronized multilingual edits, with updates cached to avoid detriment to general capabilities. Across Bi-ZsRE and MzsRE benchmarks, LU-LAFNs achieves state-of-the-art reliability, generality, and locality while avoiding cross-language knowledge conflicts that plague prior methods. The study also analyzes how updated layers and the number of LAFNs, as well as locating strategies, influence edit performance, underscoring the importance of semantic coupling in multilingual knowledge editing.

Abstract

Multilingual knowledge editing (MKE) aims to simultaneously update factual knowledge across multiple languages within large language models (LLMs). Previous research indicates that the same knowledge across different languages within LLMs exhibits a degree of shareability. However, most existing MKE methods overlook the connections of the same knowledge between different languages, resulting in knowledge conflicts and limited edit performance. To address this issue, we first investigate how LLMs process multilingual factual knowledge and discover that the same factual knowledge in different languages generally activates a shared set of neurons, which we call language-agnostic factual neurons (LAFNs). These neurons represent the same factual knowledge shared across languages and imply the semantic connections among multilingual knowledge. Inspired by this finding, we propose a new MKE method by Locating and Updating Language-Agnostic Factual Neurons (LU-LAFNs) to edit multilingual knowledge simultaneously, which avoids knowledge conflicts and thus improves edit performance. Experimental results on Bi-ZsRE and MzsRE benchmarks demonstrate that our method achieves the best edit performance, indicating the effectiveness and importance of modeling the semantic connections among multilingual knowledge.

Figures (3)

• Figure 1: The identified neuron numbers in each layer of Qwen2-7B and Llama-3.1-8B. "xxx-en" and "xxx-zh" represent the English and Chinese factual neurons respectively. "xxx-LAFNs" refers to the language-agnostic factual neurons shared by English and Chinese.
• Figure 2: The architecture of our LU-LAFNs. Given the multilingual knowledge to be edited (including the aligned multilingual subject set $S_G$), we first locate the corresponding language-agnostic factual neurons $D_G$. Then the update values $\Delta V_{D_G}$ is optimized for modifying $D_G$, and $\{S_G: \Delta V_{D_G}\}$ is stored in cache. When the subject of the user query is matched in the cache, the relative $\Delta V_{D_G}$ is used for model inference.
• Figure 3: The average EM (%) results of four metrics (Reliability, Generality, Locality, and Portability) on the MzsRE dataset using Llama-3.1-8B, Qwen2-7B, and bloomz-7b1-mt as backbones. Values below 40.0 are shown in the same light color, while higher values have deeper colors indicating better performance. The orange box highlights the mean results across 12 languages. The detailed results for each metric are listed in Appendix \ref{['sec:appendix-12langs']}.