CLM-Bench: Benchmarking and Analyzing Cross-lingual Misalignment of LLMs in Knowledge Editing

TL;DR

CLM-Bench tackles cross-lingual misalignment in knowledge editing by introducing a culture-aware Chinese-first CounterFact-derived benchmark aligned with English. The study demonstrates that language-specific edits reside in nearly orthogonal subspaces and combine additively in mixed editing, challenging the interlingua assumption for multilingual KE. Empirical results across Llama3-8B, Qwen2-7B, Llama2-7B Chinese, and Mistral-7B with MEMIT, AlphaEdit, and PMET reveal substantial cross-lingual transfer gaps and limited synergy from mixed edits. These findings argue for culturally grounded benchmarks and provide a geometric explanation for why current cross-lingual KE methods struggle, guiding future multilingual knowledge editing research.

Abstract

Knowledge Editing (KE) has emerged as a promising paradigm for updating facts in Large Language Models (LLMs) without retraining. However, progress in Multilingual Knowledge Editing (MKE) is currently hindered by biased evaluation frameworks. We observe that existing MKE benchmarks are typically constructed by mechanically translating English-centric datasets into target languages (e.g., English-to-Chinese). This approach introduces translation artifacts and neglects culturally specific entities native to the target language, failing to reflect the true knowledge distribution of LLMs. To address this, we propose CLM-Bench, a culture-aware benchmark constructed using a native Chinese-first methodology. We curate 1,010 high-quality CounterFact pairs rooted in Chinese cultural contexts and align them with English counterparts. Using CLM-Bench, we conduct extensive experiments on representative LLMs (e.g., Llama-3, Qwen2) and reveal a significant Cross-lingual Misalignment: edits in one language function independently and fail to propagate to the other. We further provide a geometric explanation via layer-wise representation analysis, demonstrating that edit vectors for Chinese and English are nearly orthogonal -- residing in disjoint subspaces -- while mixed-lingual editing exhibits linear additivity of these vectors. Our findings challenge the effectiveness of current methods in cross-lingual transfer and underscore the importance of culturally native benchmarks.

Figures (8)

• Figure 1: This figure illustrates Chinese–English editing independence: Applying the same edit with different target values yields different Chinese and English outputs.
• Figure 2: Overview of CLM-Bench, covering 6 subdomains and 24 domains.
• Figure 3: Geometric analysis of language-specific edit vectors across batch sizes. Top: Independence metrics show Chinese and English edits are nearly orthogonal. Bottom: Additivity metrics demonstrate mixed edits closely approximate the linear sum of monolingual edits. Bar charts compare independence (blue) versus additivity (orange), confirming orthogonality between languages and linear composability in mixed editing.
• Figure 4: Cross-lingual edit statistics across layers 9--12, including change distributions, neuron comparisons, CDFs, overlap, and clustering structure.
• Figure 5: Language-specific patterns across layers 9--12, including delta cosine similarity (top-left), projection difference (top-right), bias correlation (bottom-left), and raw projections (bottom-right).