Disentangling Knowledge Representations for Large Language Model Editing

TL;DR

This work tackles the challenge of updating embedded knowledge in large language models without degrading fine-grained irrelevant facts that share the edited subject. It introduces DiKE, a two-component framework consisting of Knowledge Representation Disentanglement (KRD) and Disentanglement-based Knowledge Edit (DKE), augmented by a closed-form rank-one update to enable efficient, targeted edits. KRD splits subject representations into target-related and unrelated components, while DKE updates only the related part and enforces invariance on the unrelated part to preserve fine-grained knowledge; a new dataset FINE-KED evaluates preservation across relational similarity levels. Across GPT-2 XL, GPT-J, and LLaMA3, DiKE achieves superior preservation of fine-grained irrelevant knowledge while maintaining competitive editing performance on CounterFact, outperforming baselines like ROME, MEMIT, and AlphaEdit. The results highlight the value of representation disentanglement for robust, scalable knowledge editing with implications for safer and more controllable LLM deployment.

Abstract

Knowledge Editing has emerged as a promising solution for efficiently updating embedded knowledge in large language models (LLMs). While existing approaches demonstrate effectiveness in integrating new knowledge and preserving the original capabilities of LLMs, they fail to maintain fine-grained irrelevant knowledge facts that share the same subject as edited knowledge but differ in relation and object. This challenge arises because subject representations inherently encode multiple attributes, causing the target and fine-grained irrelevant knowledge to become entangled in the representation space, and thus vulnerable to unintended alterations during editing. To address this, we propose DiKE, a novel approach that Disentangles Knowledge representations for LLM Editing (DiKE). DiKE consists of two key components: a Knowledge Representation Disentanglement (KRD) module that decomposes the subject representation into target-knowledgerelated and -unrelated components, and a Disentanglement-based Knowledge Edit (DKE) module that updates only the target-related component while explicitly preserving the unrelated one. We further derive a closed-form, rank-one parameter update based on matrix theory to enable efficient and minimally invasive edits. To rigorously evaluate fine-grained irrelevant knowledge preservation, we construct FINE-KED, a new benchmark comprising fine-grained irrelevant knowledge at different levels of relational similarity to the edited knowledge. Extensive experiments across multiple LLMs demonstrate that DiKE substantially improves fine-grained irrelevant knowledge preservation while maintaining competitive general editing performance.

Figures (9)

• Figure 1: Knowledge editing can unintentionally affect fine-grained irrelevant knowledge. Figure (a): Preservation performance on fine-grained vs. coarse-grained irrelevant knowledge. Figure (b): Illustration of knowledge editing can unintentionally affect fine-grained irrelevant knowledge.
• Figure 2: Overview of DiKE architecture. The KRD module is first trained to disentangle target-knowledge-related and -unrelated representations. The DKE module then edits the related representation to inject new knowledge while constraining the unrelated part to preserve fine-grained irrelevant knowledge. A closed-form solution (Eq.(\ref{['eq:ours-solution']})) is derived for parameter updates.(Note: The KRD module is pre-trained on the training set and does not require retraining for subsequent editing tasks)
• Figure 3: Ablation studies on FINE-KED in terms of Efficacy and Relational Locality.
• Figure 4: Performance of subject-consistent batch editing on the FINE-KED.
• Figure 5: Performance of DiKE with varying sizes of $\mathcal{N}$ in the KRD module.