Are We Evaluating the Edit Locality of LLM Model Editing Properly?
Wei Liu, Haomei Xu, Hongkai Liu, Zhiying Deng, Ruixuan Li, Heng Huang, Yee Whye Teh, Wee Sun Lee
TL;DR
The work critically examines how edit locality (specificity) is evaluated in LLM model editing and exposes serious shortcomings in ground-truth–based metrics that rely on definite answers. It introduces GT-free, continuous evaluation via distributional measures, notably $D_{KL}$ on logits and top-$k$ support overlap, and demonstrates these metrics are more sensitive and better aligned with regularization strength across multiple models and datasets. Through extensive experiments, the authors show GT-free metrics enable finer discrimination of non-target knowledge preservation and offer stable method rankings compared to traditional GT-based approaches. The proposed evaluation protocol thus provides a practical, interpretable, and robust tool for assessing knowledge preservation in open-ended LLM outputs, with implications for fairer benchmarking and more reliable model editing.
Abstract
Model editing has recently emerged as a popular paradigm for efficiently updating knowledge in LLMs. A central desideratum of updating knowledge is to balance editing efficacy, i.e., the successful injection of target knowledge, and specificity (also known as edit locality), i.e., the preservation of existing non-target knowledge. However, we find that existing specificity evaluation protocols are inadequate for this purpose. We systematically elaborated on the three fundamental issues it faces. Beyond the conceptual issues, we further empirically demonstrate that existing specificity metrics are weakly correlated with the strength of specificity regularizers. We also find that current metrics lack sufficient sensitivity, rendering them ineffective at distinguishing the specificity performance of different methods. Finally, we propose a constructive evaluation protocol. Under this protocol, the conflict between open-ended LLMs and the assumption of determined answers is eliminated, query-independent fluency biases are avoided, and the evaluation strictness can be smoothly adjusted within a near-continuous space. Experiments across various LLMs, datasets, and editing methods show that metrics derived from the proposed protocol are more sensitive to changes in the strength of specificity regularizers and exhibit strong correlation with them, enabling more fine-grained discrimination of different methods' knowledge preservation capabilities.