REPAIR: Robust Editing via Progressive Adaptive Intervention and Reintegration

TL;DR

REPAIR introduces a robust lifelong editing framework for large language models that mitigates instability and unintended ripple effects from sequential edits. It combines a dual-memory routing mechanism, distribution-aware inner-batch knowledge distillation, closed-loop error feedback with memory pruning, and loss-aware subspace merging via a weighted TIES operator to achieve precise, low-cost updates while preserving non-target knowledge. Empirical results across multiple model families and editing scales show improved editing accuracy, stronger generalization, and reduced hallucinations compared with prior baselines, demonstrating solid stability in large-scale sequential edits. The framework offers practical significance for continuously evolving LLMs by delivering reliable, auditable, and locality-preserving updates under realistic editing workloads.

Abstract

Post-training for large language models (LLMs) is constrained by the high cost of acquiring new knowledge or correcting errors and by the unintended side effects that frequently arise from retraining. To address these issues, we introduce REPAIR (Robust Editing via Progressive Adaptive Intervention and Reintegration), a lifelong editing framework designed to support precise and low-cost model updates while preserving non-target knowledge. REPAIR mitigates the instability and conflicts of large-scale sequential edits through a closed-loop feedback mechanism coupled with dynamic memory management. Furthermore, by incorporating frequent knowledge fusion and enforcing strong locality guards, REPAIR effectively addresses the shortcomings of traditional distribution-agnostic approaches that often overlook unintended ripple effects. Our experiments demonstrate that REPAIR boosts editing accuracy by 10%-30% across multiple model families and significantly reduces knowledge forgetting. This work introduces a robust framework for developing reliable, scalable, and continually evolving LLMs.

Figures (7)

• Figure 1: Problems and our solutions. REPAIR achieves closed-loop feedback, fine-grained knowledge integration, weighted knowledge merging, and robust editing performance.
• Figure 2: The overall structure of REPAIR. An edit, such as changing the capital of France from "Lyon" to "Paris," is stored as a parameter update, $\Delta\theta$, in the Side Memory. An Error Sample Monitor evaluates the performance of each edit ($Out_i^e$). If the error rate, $Err_{thresh}$, for an edit on a new sample exceeds a threshold $\epsilon$, the Side Memory Pruning module removes the erroneous update. The system then reintegrates new and error-prone samples for continuous learning.
• Figure 3: Average Editing Performance of WikiBigEdit Across Different Models
• Figure 4: Activation Score Visualization. Results on LLaMA‑3 for the WikiBigEdit dataset (N=1550) for the QA task and the SelfCheckGPT dataset for hallucination (N=600).
• Figure 5: Performance comparison of different components. Each radar chart shows performance on four metrics: Rel., gen., loc., and OP. on Qwen2.5 with ZsRE.

Theorems & Definitions (23)

• Definition 2.1: Lifelong Model Editing
• Lemma 1: Norm Bound under Masked Updates
• proof
• Theorem 1: Inter-Shard Stability
• proof
• Lemma 2: Linear Error Decrease
• Theorem 2: Finite-Time Convergence
• proof
• Theorem 3: Closed-Loop Stability of REPAIR
• proof : Proof Sketch