EAMET: Robust Massive Model Editing via Embedding Alignment Optimization

TL;DR

This work tackles the robustness gap in massive model editing by identifying embedding misalignment between key and residual spaces as the fundamental bottleneck. It introduces EAMET, a memory-editing framework that progressively aligns residual embeddings with the key space via KL-divergence and top-$M$ cosine-similarity-based MSE losses, while preserving preserved knowledge. Across six LLMs and three datasets, EAMET achieves high editing efficacy (around 90% for 10k edits) and demonstrates strong robustness to long prefixes and multiple edits of the same subject, outperforming prior methods like MEMIT, PMET, and ROME. The approach preserves general model capabilities, scales to large edit sets, and integrates with sequential editing to enable larger batch edits per step, offering a practical pathway for up-to-date knowledge in real-world AI systems.

Abstract

Model editing techniques are essential for efficiently updating knowledge in large language models (LLMs). However, the effectiveness of existing approaches degrades in massive editing scenarios, particularly when evaluated with practical metrics. Their robustness is also limited in context-rich settings or when editing multiple facts of the same subject simultaneously. We attribute these failures to the embedding misalignment among knowledge items, which undermines editing reliability at scale. To address this, we propose EAMET (Embedding Alignment Model Editing in Transformers), which addresses this issue by aligning the space of key and residual embeddings. Extensive experiments across six LLMs and three datasets demonstrate that EAMET consistently outperforms existing methods, achieving about 90\% editing efficacy when editing 10k facts. Codes and datasets are publicly available at https://ybdai7.github.io/eamet-page/.

Figures (10)

• Figure 1: Illustration of current methods and our proposed EAMET in evaluating massive editing. Here, "[PREFIX] Sentence" and "{Sentence $\mid$$s_i=\text{Jeep Commander}$}" denote the scenarios where the edited knowledge is preceded by prefixes and where multiple facts share the same subject, respectively.
• Figure 2: Illustration of the model editing problem.
• Figure 3: Impact of editing same-subject samples. Shaded region indicates shared items.
• Figure 4: Method Overview of EAMET.
• Figure 5: Editing performance of different methods across varying prefix lengths.