SWEA: Updating Factual Knowledge in Large Language Models via Subject Word Embedding Altering

TL;DR

This work tackles the challenge of updating factual knowledge in large language models without costly retraining. It introduces SWEA⊕OS, a detachable, input-embedding–level editing approach that uses token-level matching to identify subjects and an Optimizing then Suppressing (OS) fusion to generate editing embeddings by suppressing subject-specific KEDs. The method achieves state-of-the-art performance on CounterFact, zsRE, and RippleEdits, while preserving inference cost and offering strong stability in batch and sequential editing scenarios. By enabling efficient, targeted updates through subject-word embedding alterations, SWEA⊕OS provides a practical pathway for maintaining up-to-date model knowledge with reduced resource demands.

Abstract

The general capabilities of large language models (LLMs) make them the infrastructure for various AI applications, but updating their inner knowledge requires significant resources. Recent model editing is a promising technique for efficiently updating a small amount of knowledge of LLMs and has attracted much attention. In particular, local editing methods, which directly update model parameters, are proven suitable for updating small amounts of knowledge. Local editing methods update weights by computing least squares closed-form solutions and identify edited knowledge by vector-level matching in inference, which achieve promising results. However, these methods still require a lot of time and resources to complete the computation. Moreover, vector-level matching lacks reliability, and such updates disrupt the original organization of the model's parameters. To address these issues, we propose a detachable and expandable Subject Word Embedding Altering (SWEA) framework, which finds the editing embeddings through token-level matching and adds them to the subject word embeddings in Transformer input. To get these editing embeddings, we propose optimizing then suppressing fusion method, which first optimizes learnable embedding vectors for the editing target and then suppresses the Knowledge Embedding Dimensions (KEDs) to obtain final editing embeddings. We thus propose SWEA$\oplus$OS method for editing factual knowledge in LLMs. We demonstrate the overall state-of-the-art (SOTA) performance of SWEA$\oplus$OS on the CounterFact and zsRE datasets. To further validate the reasoning ability of SWEA$\oplus$OS in editing knowledge, we evaluate it on the more complex RippleEdits benchmark. The results demonstrate that SWEA$\oplus$OS possesses SOTA reasoning ability.

Figures (6)

• Figure 1: Difference between our method and existing local editing methods. Our method focuses on altering the word embedding for the input via token-level matching, while existing local editing methods edit Feed Forward Network (FFN) and identify editing knowledge by vector-level matching. Mismatching is more likely to occur in vector-level matching, which leads to erroneous recognition of editing knowledge.
• Figure 2: Overview of SWEA$\oplus$OS. In fusion stage, we first optimize a learnable embedding vector for target knowledge "Currently, Nvidia's most advanced GPU is the B200." Second, using knowledge attribution method, we find the KEDs of 'Nvidia' regarding "its most advanced GPU". Finally, we fuse the optimized embedding vector with these KEDs subtracted to obtain the editing embeddings. In inference stage, we add these editing embeddings to the embedding of the subject 'Nvidia' for inference.
• Figure 3: Results of sequential batch editing of SWEA$\oplus$OS, PMET, MEMIT, and ROME. To better display the results, we divide the fluency by the original fluency (i.e., 622.4) and then multiply by 100 to make it fall between 0 and 100.
• Figure 4: Editing results of SWEA$\oplus$OS and baselines with the number of edits (X-axis) on COUNTERFACT dataset.
• Figure 5: The average inference time of the original model and the model under the SWEA framework on the CounterFact and zsRE editing instances.