Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UNLEARN Efficient Removal of Knowledge in Large Language Models

Tyler Lizzo, Larry Heck

TL;DR

This work tackles the problem of forgetting knowledge in large language models without retraining by introducing UNLEARN, a subspace-based method that identifies and discriminates task-specific information to enable targeted unlearning. The framework comprises subspace identification, discrimination, and removal, yielding strong forgetting (≈96%) on targeted tasks while preserving other tasks with minimal degradation, and extends to a dual learning mode LEARN that adds knowledge with comparable efficacy to LoRA. Empirical results on Llama 2 70B demonstrate superior discrimination over prior methods across diverse benchmarks, with additional validation on LEARN showing substantial improvements on LegalBench. The proposed approach offers a general, efficient pathway for privacy-preserving model adaptation and task-specific optimization in multi-task settings.

Abstract

Given the prevalence of large language models (LLMs) and the prohibitive cost of training these models from scratch, dynamically forgetting specific knowledge e.g., private or proprietary, without retraining the model has become an important capability. This paper proposes a novel method to achieve this objective called UNLEARN. The approach builds upon subspace methods to identify and specifically target the removal of knowledge without adversely affecting other knowledge in the LLM. Results demonstrate 96% of targeted knowledge can be forgotten while maintaining performance on other knowledge within 2.5% of the original model, significantly outperforming the discriminatory abilities of the previous state-of-the-art. A dual method called LEARN is also proposed for targeted knowledge addition. Results show LEARN can match the fine-tuning accuracy of Low-Rank Adaptation (LoRA) without adversely affecting similar tasks.

UNLEARN Efficient Removal of Knowledge in Large Language Models

TL;DR

This work tackles the problem of forgetting knowledge in large language models without retraining by introducing UNLEARN, a subspace-based method that identifies and discriminates task-specific information to enable targeted unlearning. The framework comprises subspace identification, discrimination, and removal, yielding strong forgetting (≈96%) on targeted tasks while preserving other tasks with minimal degradation, and extends to a dual learning mode LEARN that adds knowledge with comparable efficacy to LoRA. Empirical results on Llama 2 70B demonstrate superior discrimination over prior methods across diverse benchmarks, with additional validation on LEARN showing substantial improvements on LegalBench. The proposed approach offers a general, efficient pathway for privacy-preserving model adaptation and task-specific optimization in multi-task settings.

Abstract

Given the prevalence of large language models (LLMs) and the prohibitive cost of training these models from scratch, dynamically forgetting specific knowledge e.g., private or proprietary, without retraining the model has become an important capability. This paper proposes a novel method to achieve this objective called UNLEARN. The approach builds upon subspace methods to identify and specifically target the removal of knowledge without adversely affecting other knowledge in the LLM. Results demonstrate 96% of targeted knowledge can be forgotten while maintaining performance on other knowledge within 2.5% of the original model, significantly outperforming the discriminatory abilities of the previous state-of-the-art. A dual method called LEARN is also proposed for targeted knowledge addition. Results show LEARN can match the fine-tuning accuracy of Low-Rank Adaptation (LoRA) without adversely affecting similar tasks.
Paper Structure (23 sections, 8 equations, 1 figure, 3 tables)

This paper contains 23 sections, 8 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Parameter Efficient Fine-Tuning
  4. Machine Unlearning
  5. LLM Unlearning
  6. UNLEARN Method
  7. Subspace Identification
  8. Subspace Discrimination
  9. Task Removal
  10. Experiments
  11. Datasets
  12. Single Task Removal
  13. Task Discrimination
  14. Optimal Rank
  15. Using $\hbox{UNLEARN}$ to LEARN
  16. ...and 8 more sections

Figures (1)

  • Figure 1: The Subspace Identification Process. The process begins by randomly initializing the model weights and then freezing them. Then an iterative process of unfreezing, training, and refreezing each layer occurs. This results in a set of matrices that capture an accurate representation for that task.