Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMs

TL;DR

This work tackles lifelong knowledge editing in LLMs by addressing error accumulation from coarse updates. It introduces NMKE, a fine-grained framework that combines neuron-level attribution with entropy-guided dynamic sparse masking to identify and selectively modify knowledge-general and knowledge-specific neurons in the FFN. Empirical results across thousands of edits show NMKE achieves higher edit success while preserving general capabilities and internal parameter stability better than existing methods. The approach enables robust, scalable lifelong editing with minimal parameter changes and practical performance benefits for real-world knowledge maintenance.

Abstract

Lifelong knowledge editing enables continuous, precise updates to outdated knowledge in large language models (LLMs) without computationally expensive full retraining. However, existing methods often accumulate errors throughout the editing process, causing a gradual decline in both editing accuracy and generalization. To tackle this problem, we propose Neuron-Specific Masked Knowledge Editing (NMKE), a novel fine-grained editing framework that combines neuron-level attribution with dynamic sparse masking. Leveraging neuron functional attribution, we identify two key types of knowledge neurons, with knowledge-general neurons activating consistently across prompts and knowledge-specific neurons activating to specific prompts. NMKE further introduces an entropy-guided dynamic sparse mask, locating relevant neurons to the target knowledge. This strategy enables precise neuron-level knowledge editing with fewer parameter modifications. Experimental results from thousands of sequential edits demonstrate that NMKE outperforms existing methods in maintaining high editing success rates and preserving model general capabilities in lifelong editing.

Figures (14)

• Figure 1: Left: Comparison of layer-level editing with our neuron-level editing method, NMKE, highlighting the impact on Multilayer Perceptron (MLP) layers. Right: Average editability score (including edit success rate hartvigsen2023aging, generalization success rate and localization success rate zhang2024comprehensive) and average general capability (across MMLU MMLU, GSM8K gsm8k, CommonsenseQA commonsenseqa, and BBH-Zeroshot bbh) across different edit steps on LLaMA3-8B-Instruct llama3.
• Figure 2: Left: The effect of neuron masking on multiple-choice task performance. Right: Performance changes following the masking of different neuron types on the MMLU sub-task, with general neuron masking leading to a substantial performance decline, particularly in college biology accuracy, whereas domain-specific neurons cause minimal degradation. $\downarrow$ represents the accuracy drop, with the magnitude shown as a percentage.
• Figure 3: Overview of NMKE. Neuron attribution and dynamic sparse masking to selectively update Knowledge-General and Knowledge-Specific neurons, ensuring precise knowledge editing while preserving the model’s general capabilities.
• Figure 4: Generalization performance of LLaMA-3-8B-Instruct on MMLU MMLU, GSM8K gsm8k, CommonsenseQA commonsenseqa, BBH-Zeroshot bbh, and HumanEval HumanEval benchmarks after 2000 steps of sequential editing on the ZsRE (top) and CounterFact (bottom) datasets.
• Figure 5: Visualization of the distributional shift of down-projection weights in the 8th MLP layer for (a) the pre-edited LLaMA-3-8B-Instruct, (b) AlphaEdit, and (c) NMKE after $2000$ sequential edits.