Language Anisotropic Cross-Lingual Model Editing

TL;DR

This paper defines cross-lingual model editing for multilingual pre-trained language models, showing that edits in one language must propagate to others while preserving unrelated knowledge. It introduces a parallel-corpus–based framework to adapt existing monolingual editors to the cross-lingual setting and proposes language anisotropic editing via language-specific masks to target language-dependent parameter subsets. Empirical results on mLAMA and XNLI demonstrate that cross-lingual training significantly improves transferability and reduces language variability, while LiME further boosts performance and narrows language gaps, with analyses revealing learned masks capture language-specific anisotropy. The work highlights practical implications for maintaining consistent behavior across languages and provides code for reproducibility, while acknowledging limits related to parallel data availability and continual learning challenges.

Abstract

Multilingual pre-trained language models can learn task-specific abilities or memorize facts across multiple languages but inevitably make undesired predictions with specific inputs. Under similar observation, model editing aims to post-hoc calibrate a model targeted to specific inputs with keeping the model's raw behavior. However, existing work only studies the monolingual scenario, which lacks the cross-lingual transferability to perform editing simultaneously across languages. In this work, we focus on cross-lingual model editing. Firstly, we define the cross-lingual model editing task and corresponding metrics, where an edit in one language propagates to the others. Next, we propose a framework to naturally adapt monolingual model editing approaches to the cross-lingual scenario using parallel corpus. Further, we propose language anisotropic editing to improve cross-lingual editing by amplifying different subsets of parameters for each language. On the newly defined cross-lingual model editing task, we empirically demonstrate the failure of monolingual baselines in propagating the edit to multiple languages and the effectiveness of the proposed language anisotropic model editing. Our code is publicly available at https://github.com/franklear/LiME.

Figures (7)

• Figure 1: An example of cross-lingual model editing for updating facts, where we represent facts inferred from models as sentences in the dashed boxes. The goal is to update the given fact while retaining unrelated facts. Further, cross-lingual editing requires the edit in one language (e.g., en) to affect all languages (en, zh, …).
• Figure 2: The overall framework of the proposed cross-lingual model editing. Each training step consists of two stages. The editor edits the model at first, then losses for reliability and locality are obtained from the outputs of the edited model to supervise the editor. Languages of editing/updating/retaining are randomly sampled in each training step to endow the editor with language transferability. Our novel language anisotropic model editing applies soft masks according to the editing language, which are supervised using the re-parameterized $L_0$ loss.
• Figure 3: Editing performance varies across different languages. Training editors with parallel data improves overall editing performance, while decreasing the performance variance among languages.
• Figure 4: Distribution of editing performance across languages. Language Anisotropic Model Editing (LiME for short) provides overall performance improvement and closes the performance gap across languages.
• Figure 5: Cosine similarities of learned parameters of language-specific masks. In each row, we inspect the top-$1\%$ preferred dimensions of a certain language $l$ by value, on which dimensions we calculate the cosine similarities between $l$ and every languages.