CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners

TL;DR

CaKE addresses the challenge that existing knowledge editing methods fail to generalize to multi-hop reasoning by diagnosing LLM reasoning circuits and proposing circuit-aware editing. It combines circuit-aware data generation with LoRA-based edit training to actively build and solidify updated knowledge within the model’s reasoning pathways. Empirically, CaKE yields around a 20% boost in multi-hop accuracy on MQuAKE and demonstrates superior memory efficiency and scalability compared with prior KE approaches. The work provides a circuit-centric framework for updating factual knowledge in LLMs, with strong practical implications for maintaining up-to-date and reliable reasoning in large language models.

Abstract

Knowledge Editing (KE) enables the modification of outdated or incorrect information in large language models (LLMs). While existing KE methods can update isolated facts, they often fail to generalize these updates to multi-hop reasoning tasks that rely on the modified knowledge. Through an analysis of reasoning circuits -- the neural pathways LLMs use for knowledge-based inference, we find that current layer-localized KE approaches (e.g., MEMIT, WISE), which edit only single or a few model layers, inadequately integrate updated knowledge into these reasoning pathways. To address this limitation, we present CaKE (Circuit-aware Knowledge Editing), a novel method that enhances the effective integration of updated knowledge in LLMs. By only leveraging a few curated data samples guided by our circuit-based analysis, CaKE stimulates the model to develop appropriate reasoning circuits for newly incorporated knowledge. Experiments show that CaKE enables more accurate and consistent use of edited knowledge across related reasoning tasks, achieving an average improvement of 20% in multi-hop reasoning accuracy on the MQuAKE dataset while requiring less memory than existing KE methods. We release the code and data in https://github.com/zjunlp/CaKE.

Figures (7)

• Figure 1: The current edit cannot propagate the new knowledge to the reasoning circuit for multi-hop reasoning. We propose a circuit-aware edit to improve the model's multi-hop reasoning performance involving the updated knowledge.
• Figure 2: An overview of our work.
• Figure 3: Results of the intervention on the failure cases in multi-hop reasoning of LLAMA3 and Qwen2.5.
• Figure 4: Accuracies of different number hops and edit-positions in MQuAKE-CF-3k-v2 on LLAMA3-8B-Instruct.
• Figure 5: $e_2$ and $r_2$'s logits at $t_2$ in models after different knowledge editing methods.