DAFNet: Dynamic Auxiliary Fusion for Sequential Model Editing in Large Language Models

TL;DR

This work tackles sequential model editing (SME) for large language models, addressing the need to continually rectify factual mistakes as they emerge. It introduces DAFNet, a dynamic auxiliary fusion framework with intra-editing and inter-editing attention flows to capture semantic interactions across a sequence of edits and prevent forgetting, paired with the DAFSet auxiliary dataset to learn robust meta-weights. Empirical results on multiple SME benchmarks show DAFNet achieving state-of-the-art performance in both single-turn and sequential editing, with ablations confirming the importance of the proposed attention flows and data-augmentation. The approach offers a scalable path to more reliable, generalizable, and locality-preserving edits in evolving knowledge environments, potentially reducing hallucinations in deployed LLMs.

Abstract

Recently, while large language models (LLMs) have demonstrated impressive results, they still suffer from hallucination, i.e., the generation of false information. Model editing is the task of fixing factual mistakes in LLMs; yet, most previous works treat it as a one-time task, paying little attention to ever-emerging mistakes generated by LLMs. We address the task of sequential model editing (SME) that aims to rectify mistakes continuously. A Dynamic Auxiliary Fusion Network (DAFNet) is designed to enhance the semantic interaction among the factual knowledge within the entire sequence, preventing catastrophic forgetting during the editing process of multiple knowledge triples. Specifically, (1) for semantic fusion within a relation triple, we aggregate the intra-editing attention flow into auto-regressive self-attention with token-level granularity in LLMs. We further leverage multi-layer diagonal inter-editing attention flow to update the weighted representations of the entire sequence-level granularity. (2) Considering that auxiliary parameters are required to store the knowledge for sequential editing, we construct a new dataset named \textbf{DAFSet}, fulfilling recent, popular, long-tail and robust properties to enhance the generality of sequential editing. Experiments show DAFNet significantly outperforms strong baselines in single-turn and sequential editing. The usage of DAFSet also consistently improves the performance of other auxiliary network-based methods in various scenarios

Figures (6)

• Figure 1: The comparison between different model editing scenarios for LLMs including single-turn and sequential editing with $T$ steps. The single-turn editing model only edits one fact into the LLM at a time. In the sequential editing scenario, it requires to edit a series of facts continually (Best viewed in color).
• Figure 2: Statistical results of the collected DAFSet.
• Figure 3: Model overview. Our DAFNet model mainly includes three steps: Gradient Editing Signal Acquisition, Dynamic Auxiliary Fusion and Editing Training. Particularly, Dynamic Auxiliary Fusion is designed with intra and inter-editing attention flows to capture the interaction between editing facts.
• Figure 4: The influence of different properties in DAFSet for editing results. "O", "LT", "Re", "P" and "Ro" indicate original, long-tailness, recency, popularity and robustness data, respectively.
• Figure 5: The window size influence of intra-editing attention flow.