Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Editing Factual Knowledge and Explanatory Ability of Medical Large Language Models

Derong Xu, Ziheng Zhang, Zhihong Zhu, Zhenxi Lin, Qidong Liu, Xian Wu, Tong Xu, Wanyu Wang, Yuyang Ye, Xiangyu Zhao, Enhong Chen, Yefeng Zheng

TL;DR

This work proposes MedLaSA, a novel Layer-wise Scalable Adapter strategy for medical model editing that harnesses the strengths of both adding extra parameters and locate-then-edit methods for medical model editing and incorporates scalable adapters into the dense layers of LLMs.

Abstract

Model editing aims to precisely alter the behaviors of large language models (LLMs) in relation to specific knowledge, while leaving unrelated knowledge intact. This approach has proven effective in addressing issues of hallucination and outdated information in LLMs. However, the potential of using model editing to modify knowledge in the medical field remains largely unexplored, even though resolving hallucination is a pressing need in this area. Our observations indicate that current methods face significant challenges in dealing with specialized and complex knowledge in medical domain. Therefore, we propose MedLaSA, a novel Layer-wise Scalable Adapter strategy for medical model editing. MedLaSA harnesses the strengths of both adding extra parameters and locate-then-edit methods for medical model editing. We utilize causal tracing to identify the association of knowledge in neurons across different layers, and generate a corresponding scale set from the association value for each piece of knowledge. Subsequently, we incorporate scalable adapters into the dense layers of LLMs. These adapters are assigned scaling values based on the corresponding specific knowledge, which allows for the adjustment of the adapter's weight and rank. The more similar the content, the more consistent the scale between them. This ensures precise editing of semantically identical knowledge while avoiding impact on unrelated knowledge. To evaluate the editing impact on the behaviours of LLMs, we propose two model editing studies for medical domain: (1) editing factual knowledge for medical specialization and (2) editing the explanatory ability for complex knowledge. We build two novel medical benchmarking datasets and introduce a series of challenging and comprehensive metrics. Extensive experiments on medical LLMs demonstrate the editing efficiency of MedLaSA, without affecting unrelated knowledge.

Editing Factual Knowledge and Explanatory Ability of Medical Large Language Models

TL;DR

This work proposes MedLaSA, a novel Layer-wise Scalable Adapter strategy for medical model editing that harnesses the strengths of both adding extra parameters and locate-then-edit methods for medical model editing and incorporates scalable adapters into the dense layers of LLMs.

Abstract

Model editing aims to precisely alter the behaviors of large language models (LLMs) in relation to specific knowledge, while leaving unrelated knowledge intact. This approach has proven effective in addressing issues of hallucination and outdated information in LLMs. However, the potential of using model editing to modify knowledge in the medical field remains largely unexplored, even though resolving hallucination is a pressing need in this area. Our observations indicate that current methods face significant challenges in dealing with specialized and complex knowledge in medical domain. Therefore, we propose MedLaSA, a novel Layer-wise Scalable Adapter strategy for medical model editing. MedLaSA harnesses the strengths of both adding extra parameters and locate-then-edit methods for medical model editing. We utilize causal tracing to identify the association of knowledge in neurons across different layers, and generate a corresponding scale set from the association value for each piece of knowledge. Subsequently, we incorporate scalable adapters into the dense layers of LLMs. These adapters are assigned scaling values based on the corresponding specific knowledge, which allows for the adjustment of the adapter's weight and rank. The more similar the content, the more consistent the scale between them. This ensures precise editing of semantically identical knowledge while avoiding impact on unrelated knowledge. To evaluate the editing impact on the behaviours of LLMs, we propose two model editing studies for medical domain: (1) editing factual knowledge for medical specialization and (2) editing the explanatory ability for complex knowledge. We build two novel medical benchmarking datasets and introduce a series of challenging and comprehensive metrics. Extensive experiments on medical LLMs demonstrate the editing efficiency of MedLaSA, without affecting unrelated knowledge.
Paper Structure (25 sections, 8 equations, 5 figures, 7 tables)

This paper contains 25 sections, 8 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Prelimimaries
  5. Causal Tracing
  6. MedLaSA
  7. Scaling $\alpha$.
  8. Scaling Rank $r$.
  9. Medical Model Editing Benchmarks
  10. Efficacy and Generality Data Construction
  11. Medical Counter Fact Dataset.
  12. Medical Fact Explanation Dataset.
  13. Locality Data Construction.
  14. Experiments
  15. Experimental Setup
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Examples of specialized and complex knowledge in the medical domain. The red/blue symbols and background colors represent LLMs' state and output before/after model editing.
  • Figure 2: The overview of our MedLaSA. We demonstrate the process of inputting editing, rephrased, and unrelated knowledge.
  • Figure 3: Knowledge type distribution of MedCF (Left) and MedFE (Right) dataset.
  • Figure 4: Analysis of hyper-parameters $r_o$ and $\alpha_o$.
  • Figure 5: Cases of causal tracing on MedCF. The presented red/green heatmaps illustrate the impact of Attn/MLP weight restoration after corrupting input.