Beyond Memorization: A Rigorous Evaluation Framework for Medical Knowledge Editing

TL;DR

This work tackles the challenge of updating medical knowledge in large language models without full retraining by introducing MedEditBench, a rigorous medical knowledge editing evaluation framework. It constructs MedExQAedit and MedMCQAedit benchmarks and compares three editing paradigms—GTA-Edit, RE-Edit, and the proposed SGR-Edit—finding that traditional editing yields superficial memorization and poor generalization. SGR-Edit, which uses model-generated rationales as the editing target, consistently improves editing performance and interpretability across multiple base models, while enabling more robust sequential edits. The study also analyzes how sequential edits affect internal and external knowledge and general capabilities, offering practical guidance for safely deploying explainable KE in real-world medical settings and highlighting limitations and avenues for future work.

Abstract

Recently, knowledge editing (KE) has emerged as a promising approach to update specific facts in Large Language Models (LLMs) without the need for full retraining. Despite the effectiveness in general-domain benchmarks, their applicability to complex medical domain remains largely unexplored. Medical knowledge editing is particularly challenging, as it requires LLMs to internalize the knowledge and generalize to unseen scenarios for effective and interpretable decision-making. In this work, we propose a novel framework called MedEditBench to rigorously evaluate the effectiveness of existing KE methods in the medical domain. In MedEditBench, we introduce a new medical knowledge editing benchmark as well as three different knowledge editing paradigms, which are designed to assess the impact of different knowledge sources for editing. Our findings indicate that current KE methods result in only superficial memorization of the injected information, failing to generalize to new scenarios. To overcome this limitation, we present Self-Generated Rationale Editing (SGR-Edit), which utilizes model-derived rationales as the target knowledge for editing, thereby uncovering the underlying reasoning process and demonstrating significant improvements over existing KE approaches. Additionally, we offer deeper insights into medical knowledge editing, including the localization of medical knowledge in LLMs and the impact of sequential editing on evolving knowledge. This could provide practical guidance for implementing KE methods in real-world medical applications.

Figures (16)

• Figure 1: Illustration of medical knowledge editing.
• Figure 2: Overview of proposed medical knowledge editing evaluation framework (MedEditBench). Datasets are first constructed for medical knowledge editing (yellow rectangle). Three editing paradigms (GTA-Edit, RE-Edit, and SGR-Edit) are applied to evaluate the effectiveness of different target knowledge for medical knowledge editing (purple rectangle). In particular, the proposed SGR-Edit generates a rationale by prompting the LLM itself, given a QA and a reference (green rectangle). Then, the knowledge is used to edit the base LLM using editing methods like ROME (blue rectangle). In the final evaluation stage (orange rectangle), the edited LLM is evaluated using efficacy, generalization, and retention, as well as interpretability of the generated rationales. To examine how sequential medical edits affect the edited LLM’s broader knowledge and general abilities, we complement MedEditBench with external evaluations, including multi-domain knowledge (MMLU), and commonsense reasoning (HellaSwag), factual recall (TriviaQA), truthfulness (TruthfulQA), and mathematical reasoning (GSM8K).
• Figure 3: Medical knowledge editing with various editing paradigms.
• Figure 4: Interpretability comparison for SGR-Edit and RE-Edit. Quality of Reasoning (QoR) is a normalized (0–1) score via an AI judge (DeepSeek-V3 deepseek), assessing the generated rationales across five dimensions: factual accuracy, logical flow, relevance, completeness, and answer correctness (see prompt in \ref{['prompt:interpre']}).
• Figure 5: Sequential editing performance on the internal benchmark.