Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowledge Editing in Language Models via Adapted Direct Preference Optimization

Amit Rozner, Barak Battash, Lior Wolf, Ofir Lindenbaum

TL;DR

This work introduces Knowledge Direct Preference Optimization (KDPO), a variation of the Direct Preference Optimization (DPO) that is more effective for knowledge modifications and proposes treating KE as an LLM alignment problem.

Abstract

Large Language Models (LLMs) can become outdated over time as they may lack updated world knowledge, leading to factual knowledge errors and gaps. Knowledge Editing (KE) aims to overcome this challenge using weight updates that do not require expensive retraining. We propose treating KE as an LLM alignment problem. Toward this goal, we introduce Knowledge Direct Preference Optimization (KDPO), a variation of the Direct Preference Optimization (DPO) that is more effective for knowledge modifications. Our method is based on an online approach that continually updates the knowledge stored in the model. We use the current knowledge as a negative sample and the new knowledge we want to introduce as a positive sample in a process called DPO. We also use teacher-forcing for negative sample generation and optimize using the positive sample, which helps maintain localized changes. We tested our KE method on various datasets and models, comparing it to several cutting-edge methods, with 100 and 500 sequential edits. Additionally, we conducted an ablation study comparing our method to the standard DPO approach. Our experimental results show that our modified DPO method allows for more refined KE, achieving similar or better performance compared to previous methods.

Knowledge Editing in Language Models via Adapted Direct Preference Optimization

TL;DR

This work introduces Knowledge Direct Preference Optimization (KDPO), a variation of the Direct Preference Optimization (DPO) that is more effective for knowledge modifications and proposes treating KE as an LLM alignment problem.

Abstract

Large Language Models (LLMs) can become outdated over time as they may lack updated world knowledge, leading to factual knowledge errors and gaps. Knowledge Editing (KE) aims to overcome this challenge using weight updates that do not require expensive retraining. We propose treating KE as an LLM alignment problem. Toward this goal, we introduce Knowledge Direct Preference Optimization (KDPO), a variation of the Direct Preference Optimization (DPO) that is more effective for knowledge modifications. Our method is based on an online approach that continually updates the knowledge stored in the model. We use the current knowledge as a negative sample and the new knowledge we want to introduce as a positive sample in a process called DPO. We also use teacher-forcing for negative sample generation and optimize using the positive sample, which helps maintain localized changes. We tested our KE method on various datasets and models, comparing it to several cutting-edge methods, with 100 and 500 sequential edits. Additionally, we conducted an ablation study comparing our method to the standard DPO approach. Our experimental results show that our modified DPO method allows for more refined KE, achieving similar or better performance compared to previous methods.
Paper Structure (27 sections, 8 equations, 4 figures, 7 tables, 1 algorithm)

This paper contains 27 sections, 8 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Model Fine Tuning
  4. Knowledge Editing
  5. Method
  6. Preliminary and Notations
  7. Knowledge Editing as an Alignment Problem
  8. Knowledge Direct Preference Optimization
  9. Generation Step
  10. Optimization Step
  11. Experiments
  12. Evaluation Metrics
  13. Datasets
  14. Knowledge Editing Impact on General LLM Tasks
  15. Results
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Illustration of the two generation methods described in this paper. Words in red are generated based on the previous sequence. In our teacher forcing method, we use the ground truth (in blue) instead of the prediction. This generation will show gains at the optimization step.
  • Figure 2: Illustration of an optimization cycle. We note tokens that do not affect the loss by the "prohibited" sign. The arrows indicate which tokens DPO and KDPO objectives will increase/decrease their log-prob. Note that in KDPO, the "Italy" token is not optimized because it is the same in both sequences, which cancels out the two loss terms of the objective.
  • Figure 3: Comparative result for the four metrics in ZsRE datasets using algorithms discussed in this paper. Fluency results were scaled with a factor of 10 for better visibility.
  • Figure 4: Comparative result for the four metrics averaged over all datasets in Tab. \ref{['table:small_lm']} using algorithms discussed in this section on Qwen1.5-0.5B with 500 sequential edits. Fluency results were scaled with a factor of 10 for better visibility.