Multiplicative Orthogonal Sequential Editing for Language Models

TL;DR

This work tackles sequential knowledge editing in large language models by diagnosing instability caused by additive updates and proposing MOSE, a multiplicative editing paradigm that uses left-multiplication by an orthogonal update to preserve numerical stability. The authors prove theoretically that orthogonal transformations maintain both the Frobenius norm and the condition number, and they formulate a Procrustes-based solution to compute the optimal orthogonal update. Empirically, MOSE outperforms six known edits methods across three LLMs and multiple datasets, achieving a 12.08% improvement in sequential editing while retaining about 95.73% of general abilities, and it remains robust in batch-edit scenarios. The approach demonstrates practical potential for scalable, stable continual knowledge integration in LLMs, with code released for reproducibility.

Abstract

Knowledge editing aims to efficiently modify the internal knowledge of large language models (LLMs) without compromising their other capabilities. The prevailing editing paradigm, which appends an update matrix to the original parameter matrix, has been shown by some studies to damage key numerical stability indicators (such as condition number and norm), thereby reducing editing performance and general abilities, especially in sequential editing scenario. Although subsequent methods have made some improvements, they remain within the additive framework and have not fundamentally addressed this limitation. To solve this problem, we analyze it from both statistical and mathematical perspectives and conclude that multiplying the original matrix by an orthogonal matrix does not change the numerical stability of the matrix. Inspired by this, different from the previous additive editing paradigm, a multiplicative editing paradigm termed Multiplicative Orthogonal Sequential Editing (MOSE) is proposed. Specifically, we first derive the matrix update in the multiplicative form, the new knowledge is then incorporated into an orthogonal matrix, which is multiplied by the original parameter matrix. In this way, the numerical stability of the edited matrix is unchanged, thereby maintaining editing performance and general abilities. We compared MOSE with several current knowledge editing methods, systematically evaluating their impact on both editing performance and the general abilities across three different LLMs. Experimental results show that MOSE effectively limits deviations in the edited parameter matrix and maintains its numerical stability. Compared to current methods, MOSE achieves a 12.08% improvement in sequential editing performance, while retaining 95.73% of general abilities across downstream tasks. The code is available at https://github.com/famoustourist/MOSE.

Figures (22)

• Figure 1: (a) Comparison between the previous methods and MOSE. The previous methods performed updates by adding an update matrix, whereas the MOSE employs left-multiplication by an orthogonal update matrix. (b) Comparison of editing performance after additive-based editing and after multiplicative-based editing by MOSE. (c) Comparison of general downstream task performance before editing, after additive-based editing, and after multiplicative-based editing by MOSE.
• Figure 2: Illustration of the change of Frobenius norm and condition number in sequential editing at the edited layer using additive-based methods and orthogonal transformations. We selected LLaMA3-8B and CounterFact for experiments.
• Figure 3: Edited on the CounterFact dataset, the general task performance of varying methods with LLaMA3-8B as the number of edits increases, under the single-sequential scenario.
• Figure 4: Ablation analysis of editing performance in the batch-sequential scenario. The experiment was conducted with LLaMA3-8B on CounterFact dataset.
• Figure 5: The t-SNE visualization of the hidden states of the LLaMA3-8B before and after editing, where (a) corresponds to edits made using RECT, and (b) corresponds to edits made using MOSE. The top and right curves show the marginal distributions of the two t-SNE results.