Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Better Call SAUL: Fluent and Consistent Language Model Editing with Generation Regularization

Mingyang Wang, Lukas Lange, Heike Adel, Jannik Strötgen, Hinrich Schütze

TL;DR

SAUL, a streamlined model editing method that uses sentence concatenation with augmented random facts for generation regularization, is proposed, a practical and reliable solution for model editing outperforming state-of-the-art methods while maintaining generation quality and reducing computational overhead.

Abstract

To ensure large language models contain up-to-date knowledge, they need to be updated regularly. However, model editing is challenging as it might also affect knowledge that is unrelated to the new data. State-of-the-art methods identify parameters associated with specific knowledge and then modify them via direct weight updates. However, these locate-and-edit methods suffer from heavy computational overhead and lack theoretical validation. In contrast, directly fine-tuning the model on requested edits affects the model's behavior on unrelated knowledge, and significantly damages the model's generation fluency and consistency. To address these challenges, we propose SAUL, a streamlined model editing method that uses sentence concatenation with augmented random facts for generation regularization. Evaluations on three model editing benchmarks show that SAUL is a practical and reliable solution for model editing outperforming state-of-the-art methods while maintaining generation quality and reducing computational overhead.

Better Call SAUL: Fluent and Consistent Language Model Editing with Generation Regularization

TL;DR

SAUL, a streamlined model editing method that uses sentence concatenation with augmented random facts for generation regularization, is proposed, a practical and reliable solution for model editing outperforming state-of-the-art methods while maintaining generation quality and reducing computational overhead.

Abstract

To ensure large language models contain up-to-date knowledge, they need to be updated regularly. However, model editing is challenging as it might also affect knowledge that is unrelated to the new data. State-of-the-art methods identify parameters associated with specific knowledge and then modify them via direct weight updates. However, these locate-and-edit methods suffer from heavy computational overhead and lack theoretical validation. In contrast, directly fine-tuning the model on requested edits affects the model's behavior on unrelated knowledge, and significantly damages the model's generation fluency and consistency. To address these challenges, we propose SAUL, a streamlined model editing method that uses sentence concatenation with augmented random facts for generation regularization. Evaluations on three model editing benchmarks show that SAUL is a practical and reliable solution for model editing outperforming state-of-the-art methods while maintaining generation quality and reducing computational overhead.
Paper Structure (23 sections, 2 figures, 15 tables)

This paper contains 23 sections, 2 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Model Editing Problem Definition.
  5. Naive Fine-tuning for Model Editing.
  6. Random Fact Augmentation.
  7. Generation Regularization.
  8. Experimental Setup
  9. Datasets and Baselines.
  10. Training Details.
  11. Evaluation Metrics.
  12. Results and Analysis
  13. Overall Results.
  14. Ablation Study: Fine-tuning Paradigms.
  15. Ablation Study: Data Augmentation.
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Comparison between SAUL and prior work for model editing. Prior work causes generation repetition, as the fine-tuning loss focuses only on a few target tokens. In contrast, SAUL regularizes the model's generation with sentence concatenation. Consequently, the model can still generate fluent text after model editing.
  • Figure 2: Comparison of naive fine-tuning, fine-tuning with random augmentation, and SAUL.