Conflict-Resolving and Sharpness-Aware Minimization for Generalized Knowledge Editing with Multiple Updates

TL;DR

CoRSA presents a holistic framework for generalized knowledge editing with multiple updates by jointly addressing generalization, stability, and conflict suppression. It combines LoRA-based adapters with Direct Preference Optimization (DPO) to separate old and new knowledge, and Sharpness-Aware Minimization (SAM) to flatten the loss landscape, all secured by PCGrad to resolve gradient conflicts. Empirical results across factual benchmarks and code domains show substantial improvements in generality (up to ~12.5 percentage points over LoRA) and reduced forgetting during continual updates, with strong transfer to programming tasks. The approach scales to larger models and supports adapter merging, indicating practical applicability for maintaining up-to-date LLM knowledge with minimal retraining and preserved capabilities.

Abstract

Large language models (LLMs) rely on internal knowledge to solve many downstream tasks, making it crucial to keep them up to date. Since full retraining is expensive, prior work has explored efficient alternatives such as model editing and parameter-efficient fine-tuning. However, these approaches often break down in practice due to poor generalization across inputs, limited stability, and knowledge conflict. To address these limitations, we propose the CoRSA (Conflict-Resolving and Sharpness-Aware Minimization) training framework, a parameter-efficient, holistic approach for knowledge editing with multiple updates. CoRSA tackles multiple challenges simultaneously: it improves generalization to different input forms and enhances stability across multiple updates by minimizing loss curvature, and resolves conflicts by maximizing the margin between new and prior knowledge. Across three widely used fact editing benchmarks, CoRSA achieves significant gains in generalization, outperforming baselines with average absolute improvements of 12.42% over LoRA and 10% over model editing methods. With multiple updates, it maintains high update efficacy while reducing catastrophic forgetting by 27.82% compared to LoRA. CoRSA also generalizes to the code domain, outperforming the strongest baseline by 5.48% Pass@5 in update efficacy.

Figures (8)

• Figure 1: Overview of CoRSA.(Top) Limitations of previous work: Existing approaches often fail to resolve conflicts, show poor generalization on varied inputs and instability under multiple updates (update 1, 2 in the figure), leading to catastrophic forgetting. (Bottom) CoRSA: We address these limitations through two mechanisms: (A) Conflicting Knowledge Suppression: We explicitly suppress outdated information (red line), creating a distinct separation from the new target (green line). (B) Sharpness-Aware Minimization: We minimize the sharpness of the loss landscape, leading to better generalization and stability against future parameter updates (dashed line).
• Figure 2: Log-probabilities for new knowledge (solid lines) and old knowledge (dashed lines) during training Qwen-3-4B on MQuAKE.
• Figure 3: Trade-offs between the percentage of data used for updates and forgetting in the continual knowledge revision setting. CoRSA consistently demonstrates superior stability, achieving substantially lower forgetting rates compared to baselines across all data settings.
• Figure 4: Generality of LoRA, F-Learning, and CoRSA when updating Qwen models of different sizes (4B/8B/14B) on CounterFact, ZsRE, and MQuAKE. CoRSA consistently yields the best generality across model sizes.
• Figure 5: Sensitivity analysis of the hyperparameter $\lambda$ on the CounterFact dataset. We evaluate the Generality score across $\lambda \in \{0.2, 0.5, 1.0, 2.0, 5.0\}$. The results show that performance is maximized at $\lambda=1.0$.