Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EvoEdit: Evolving Null-space Alignment for Robust and Efficient Knowledge Editing

Sicheng Lyu, Yu Gu, Xinyu Wang, Jerry Huang, Sitao Luan, Yufei Cui, Xiao-Wen Chang, Peng Lu

TL;DR

EvoEdit tackles the challenge of updating knowledge in large language models without full retraining, addressing catastrophic interference during sequential edits. It replaces the traditional locate-then-edit approach with sequential null-space alignment, projecting each new edit into the null space of both preserved and prior edits to maintain stability. The method comes with theoretical guarantees and a computationally efficient solver, achieving competitive edits with up to 3.53x speedups in practice. Overall, EvoEdit advances principled, scalable continual knowledge updates for real-world LLM deployment.

Abstract

Large language models (LLMs) require continual updates to rectify outdated or erroneous knowledge. Model editing has emerged as a compelling paradigm for introducing targeted modifications without the computational burden of full retraining. Existing approaches are mainly based on a locate-then-edit framework. However, in sequential editing contexts, where multiple updates are applied over time, they exhibit significant limitations and suffer from catastrophic interference, i.e., new edits compromise previously integrated updates and degrade preserved knowledge. To address these challenges, we introduce EvoEdit, a novel editing strategy that mitigates catastrophic interference through sequential null-space alignment, enabling stable and efficient model editing. By performing sequential null-space alignment for each incoming edit, EvoEdit preserves both original and previously modified knowledge representations and maintains output invariance on preserved knowledge even across long edit sequences, effectively mitigating interference. Evaluations on real-world sequential knowledge-editing benchmarks show that EvoEdit achieves better or comparable performance than prior state-of-the-art locate-then-edit techniques, with up to 3.53 times speedup. Overall, these results underscore the necessity of developing more principled approaches for designing LLMs in dynamically evolving information settings, while providing a simple yet effective solution with strong theoretical guarantees.

EvoEdit: Evolving Null-space Alignment for Robust and Efficient Knowledge Editing

TL;DR

EvoEdit tackles the challenge of updating knowledge in large language models without full retraining, addressing catastrophic interference during sequential edits. It replaces the traditional locate-then-edit approach with sequential null-space alignment, projecting each new edit into the null space of both preserved and prior edits to maintain stability. The method comes with theoretical guarantees and a computationally efficient solver, achieving competitive edits with up to 3.53x speedups in practice. Overall, EvoEdit advances principled, scalable continual knowledge updates for real-world LLM deployment.

Abstract

Large language models (LLMs) require continual updates to rectify outdated or erroneous knowledge. Model editing has emerged as a compelling paradigm for introducing targeted modifications without the computational burden of full retraining. Existing approaches are mainly based on a locate-then-edit framework. However, in sequential editing contexts, where multiple updates are applied over time, they exhibit significant limitations and suffer from catastrophic interference, i.e., new edits compromise previously integrated updates and degrade preserved knowledge. To address these challenges, we introduce EvoEdit, a novel editing strategy that mitigates catastrophic interference through sequential null-space alignment, enabling stable and efficient model editing. By performing sequential null-space alignment for each incoming edit, EvoEdit preserves both original and previously modified knowledge representations and maintains output invariance on preserved knowledge even across long edit sequences, effectively mitigating interference. Evaluations on real-world sequential knowledge-editing benchmarks show that EvoEdit achieves better or comparable performance than prior state-of-the-art locate-then-edit techniques, with up to 3.53 times speedup. Overall, these results underscore the necessity of developing more principled approaches for designing LLMs in dynamically evolving information settings, while providing a simple yet effective solution with strong theoretical guarantees.
Paper Structure (52 sections, 11 theorems, 75 equations, 2 figures, 4 tables, 2 algorithms)

This paper contains 52 sections, 11 theorems, 75 equations, 2 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluation of Model Editing
  4. Taxonomy of Model Editing
  5. Preliminaries
  6. Feed-Forward Blocks as Associative Memory
  7. Model Editing in LLMs
  8. Null-space of Preserved Knowledge
  9. Null-space Projection.
  10. Null-space Projector Estimation.
  11. Null-Space Drift and Interference
  12. Methodology
  13. Efficient Null-space Alignment
  14. Sequential Editing via aligned Null-space Projector
  15. Theoretical Analysis
  16. ...and 37 more sections

Key Result

Theorem 4.1

Suppose that at each step $j$ the update uses ${\bm{Q}}_j$ as any orthonormal basis of $\mathrm{Range}\!\left({\bm{R}}_j\right)=\mathrm{Range}\!\left({\bm{P}}_{j-1}{\bm{K}}_j\right)$, i.e., no truncation is applied and all nonzero left singular directions of ${\bm{R}}_j$ are retained. Then, for all

Figures (2)

  • Figure 1: Comparison of EvoEdit and AlphaEdit on Llama-3-8B across four evaluation benchmarks (MRPC, NLI, SST, and MMLU). The experiments are conducted with a batch size of 1 over a total of 2000 sequential edits. The $x$-axis represents the cumulative number of edits applied to the model, while the $y$-axis indicates the corresponding F1 score, reflecting the effectiveness of each method in preserving and updating model knowledge over successive edits.
  • Figure 2: Rewrite and Paraphrase accuracy (%) on the first 100 edited facts under two editing methods (AlphaEdit and EvoEdit). We report performance at two stages: immediately after the first 100 edits (100 edits) and after 2000 total edits. For each stage, bars show accuracy on the Rewrite and Paraphrase probes. We use a batch size of 1. Evaluated is conducted using Llama-3-8B on Counterfact.

Theorems & Definitions (22)

  • Theorem 4.1: Exact equivalence without truncation
  • Theorem 4.2: Global error bound with truncation
  • Corollary 4.3: Interference bound
  • Remark 4.4: Numerical truncation
  • Proposition A.1
  • proof
  • Proposition A.2
  • proof
  • Proposition A.3
  • proof
  • ...and 12 more