Reversible Lifelong Model Editing via Semantic Routing-Based LoRA

Abstract

The dynamic evolution of real-world necessitates model editing within Large Language Models. While existing methods explore modular isolation or parameter-efficient strategies, they still suffer from semantic drift or knowledge forgetting due to continual updating. To address these challenges, we propose SoLA, a Semantic routing-based LoRA framework for lifelong model editing. In SoLA, each edit is encapsulated as an independent LoRA module, which is frozen after training and mapped to input by semantic routing, allowing dynamic activation of LoRA modules via semantic matching. This mechanism avoids semantic drift caused by cluster updating and mitigates catastrophic forgetting from parameter sharing. More importantly, SoLA supports precise revocation of specific edits by removing key from semantic routing, which restores model's original behavior. To our knowledge, this reversible rollback editing capability is the first to be achieved in existing literature. Furthermore, SoLA integrates decision-making process into edited layer, eliminating the need for auxiliary routing networks and enabling end-to-end decision-making process. Extensive experiments demonstrate that SoLA effectively learns and retains edited knowledge, achieving accurate, efficient, and reversible lifelong model editing.

Figures (5)

• Figure 1: Comparison of trainable parameters and ERR accuracy between SoLA (Ours) and other methods. All experiments are conducted on Scotus datasets with the same backbone.
• Figure 2: Number of mismatches in LoRA allocation with ERR/TRR accuracy in MELO. All experiments are conducted on Scotus datasets with the same backbone.
• Figure 3: Main framework of our method. a) indicates the edited layer in model, where we retain the transformer block of base model frozen and add trainable LoRA module to the edited layer of base model. b) shows the editing process, where every edit will be assigned LoRA module and the query of input will be mapped to assigned LoRA module. c) presents the matching process between query of input and LoRA module in reference. Colour green indicates trainable, while color grey is frozen.
• Figure 4: Figure(a)–(c) presents every edit accuracy on SCOTUS datasets with BERT. All experiment settings are the same as Tab.\ref{['tab:main result']}.
• Figure 5: Visualization of zsRE dataset encoder output from T5-small with t-SNE and the dots with same color and shape are input and rephrase sentence. All experiment settings are the same as Tab.\ref{['tab:main result']}.