MEMoE: Enhancing Model Editing with Mixture of Experts Adaptors

TL;DR

This work tackles the challenge of updating knowledge in large language models without degrading broad capabilities. MEMoE introduces a bypass Mixture-of-Experts adapter with a knowledge anchor routing mechanism, enabling targeted edits while preserving the base model’s parameters. The approach achieves state-of-the-art performance in both batch and sequential editing, delivering near-perfect Locality and high Generality (often >90) while maintaining strong Reliability. By routing inputs to experts based on anchored knowledge, MEMoE enhances generalization of updated knowledge with minimal impact on the model’s overall behavior, offering practical, scalable model-editing capabilities for real-world deployments. The core ideas are formalized through a gating mechanism G and a knowledge-anchor routing scheme, with the key equations h_i = W_0 x_i + λ ∑_{e=1}^{E} G_e W_e x_i and G = top_k(softmax(W_g·R(x_i) + ε)), where R_anchor(x_i) = concat(x_i, embed(anchor)).

Abstract

Model editing aims to efficiently alter the behavior of Large Language Models (LLMs) within a desired scope, while ensuring no adverse impact on other inputs. Recent years have witnessed various model editing methods been proposed. However, these methods either exhibit poor overall performance or struggle to strike a balance between generalization and locality. We propose MEMoE, a model editing adapter utilizing a Mixture of Experts (MoE) architecture with a knowledge anchor routing strategy. MEMoE updates knowledge using a bypass MoE structure, keeping the original parameters unchanged to preserve the general ability of LLMs. And, the knowledge anchor routing ensures that inputs requiring similar knowledge are routed to the same expert, thereby enhancing the generalization of the updated knowledge. Experimental results show the superiority of our approach over both batch editing and sequential batch editing tasks, exhibiting exceptional overall performance alongside outstanding balance between generalization and locality. Our code will be available.

Figures (5)

• Figure 1: Left: Apart from our MEMoE, no method achieves both high accuracy and high balance. Right: Significant room for improvement in the overall performance of current methods.
• Figure 2: The architecture of MEMoE, compared with conventional MoE. Same color denote inputs requiring same knowledge. Pentagrams symbolize knowledge anchors within the input sentences, while squares and triangles represent ordinary input tokens during editing process and generality evaluation respectively. The distribution of tokens within the FFN illustrates that knowledge anchor consolidate inputs requiring same knowledge to the same experts.
• Figure 3: Performance on general tasks of edited models using MEMoE, MEMIT and MEND, with different batch sizes for edits.
• Figure 4: Left: Performance across different numbers of experts. Middle: Performance across different target model layers. Right: Effectiveness of activating experts. All experiments are based on LLaMA2-7B, utilizing the ZsRE dataset and batch editing settings.
• Figure 5: A sample from (a) ZsRE, (b) CounterFact.