LyapLock: Bounded Knowledge Preservation in Sequential Large Language Model Editing

TL;DR

LyapLock addresses the challenge of preserving long-term knowledge during sequential large language model edits by formulating editing as a constrained long-term optimization. It leverages Lyapunov optimization with a virtual queue $Z(t)$ to decompose the problem into per-timestep updates, yielding a closed-form perturbation update that accounts for both current and past edits. The approach provides theoretical guarantees of long-term constraint satisfaction and near-optimal editing performance, and empirical results show strong gains in editing efficacy while maintaining general capabilities across multiple models and datasets. It also demonstrates compatibility with existing editing methods and scalability up to 20k edits, highlighting practical impact for robust, ongoing knowledge updates in LLMs.

Abstract

Large Language Models often contain factually incorrect or outdated knowledge, giving rise to model editing methods for precise knowledge updates. However, current mainstream locate-then-edit approaches exhibit a progressive performance decline during sequential editing, due to inadequate mechanisms for long-term knowledge preservation. To tackle this, we model the sequential editing as a constrained stochastic programming. Given the challenges posed by the cumulative preservation error constraint and the gradually revealed editing tasks, \textbf{LyapLock} is proposed. It integrates queuing theory and Lyapunov optimization to decompose the long-term constrained programming into tractable stepwise subproblems for efficient solving. This is the first model editing framework with rigorous theoretical guarantees, achieving asymptotic optimal editing performance while meeting the constraints of long-term knowledge preservation. Experimental results show that our framework scales sequential editing capacity to over 10,000 edits while stabilizing general capabilities and boosting average editing efficacy by 11.89\% over SOTA baselines. Furthermore, it can be leveraged to enhance the performance of baseline methods. Our code is released on https://github.com/caskcsg/LyapLock.

Figures (10)

• Figure 1: Comparison of preservation loss and downstream task performance of LLaMA3 llama3 during sequential editing of 10,000 samples using current methods and LyapLock (details in Sec.\ref{['setup1']}).
• Figure 2: A formal comparison between LyapLock and current methods.
• Figure 3: The F1 scores of the LLaMA3 (8B) model on the GLUE benchmark after sequentially editing 10,000 samples on the CounterFact dataset.
• Figure 4: The preservation loss changes after sequentially editing 10,000 samples on different datasets by different LLMs.
• Figure 5: The improvement in editing performance and downstream task performance of other editing methods after incorporating LyapLock, following the sequential editing of 10,000 samples on the CounterFact dataset using the LLaMA3 model.