Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LUNE: Efficient LLM Unlearning via LoRA Fine-Tuning with Negative Examples

Yezi Liu, Hanning Chen, Wenjun Huang, Yang Ni, Mohsen Imani

TL;DR

This work tackles the problem of removing targeted information from large language models without full retraining. It introduces LUNE, a LoRA-based unlearning framework that trains only low-rank adapters on negative examples while freezing the backbone, achieving targeted forgetting with significantly reduced compute. Across four datasets, LUNE matches or surpasses full-finetuning and memory-editing baselines in unlearning efficacy while preserving general utility and exhibiting strong robustness and privacy properties. The study demonstrates that negative-example supervision, combined with low-rank adaptations, provides a scalable and practical solution for privacy-preserving unlearning in production LLMs.

Abstract

Large language models (LLMs) possess vast knowledge acquired from extensive training corpora, but they often cannot remove specific pieces of information when needed, which makes it hard to handle privacy, bias mitigation, and knowledge correction. Traditional model unlearning approaches require computationally expensive fine-tuning or direct weight editing, making them impractical for real-world deployment. In this work, we introduce LoRA-based Unlearning with Negative Examples (LUNE), a lightweight framework that performs negative-only unlearning by updating only low-rank adapters while freezing the backbone, thereby localizing edits and avoiding disruptive global changes. Leveraging Low-Rank Adaptation (LoRA), LUNE targets intermediate representations to suppress (or replace) requested knowledge with an order-of-magnitude lower compute and memory than full fine-tuning or direct weight editing. Extensive experiments on multiple factual unlearning tasks show that LUNE: (I) achieves effectiveness comparable to full fine-tuning and memory-editing methods, and (II) reduces computational cost by about an order of magnitude.

LUNE: Efficient LLM Unlearning via LoRA Fine-Tuning with Negative Examples

TL;DR

This work tackles the problem of removing targeted information from large language models without full retraining. It introduces LUNE, a LoRA-based unlearning framework that trains only low-rank adapters on negative examples while freezing the backbone, achieving targeted forgetting with significantly reduced compute. Across four datasets, LUNE matches or surpasses full-finetuning and memory-editing baselines in unlearning efficacy while preserving general utility and exhibiting strong robustness and privacy properties. The study demonstrates that negative-example supervision, combined with low-rank adaptations, provides a scalable and practical solution for privacy-preserving unlearning in production LLMs.

Abstract

Large language models (LLMs) possess vast knowledge acquired from extensive training corpora, but they often cannot remove specific pieces of information when needed, which makes it hard to handle privacy, bias mitigation, and knowledge correction. Traditional model unlearning approaches require computationally expensive fine-tuning or direct weight editing, making them impractical for real-world deployment. In this work, we introduce LoRA-based Unlearning with Negative Examples (LUNE), a lightweight framework that performs negative-only unlearning by updating only low-rank adapters while freezing the backbone, thereby localizing edits and avoiding disruptive global changes. Leveraging Low-Rank Adaptation (LoRA), LUNE targets intermediate representations to suppress (or replace) requested knowledge with an order-of-magnitude lower compute and memory than full fine-tuning or direct weight editing. Extensive experiments on multiple factual unlearning tasks show that LUNE: (I) achieves effectiveness comparable to full fine-tuning and memory-editing methods, and (II) reduces computational cost by about an order of magnitude.

Paper Structure

This paper contains 36 sections, 13 equations, 4 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Related Work
  2. Machine Unlearning in Large Language Models
  3. Parameter-Efficient Fine-Tuning (PEFT)
  4. Preliminaries
  5. Background on Machine Unlearning in LLMs
  6. Problem Formulation
  7. Methodology
  8. Motivation and Overview
  9. Low-Rank Adaptation Mechanism
  10. Fine-Tuning with Negative Examples
  11. Negative Examples Construction
  12. Theoretical Insight: Low-Rank Negative Updates
  13. Complexity comparison
  14. Experiments
  15. Experimental Setup
  16. ...and 21 more sections

Figures (4)

  • Figure 1: Illustration of the LLM unlearning task. The goal is to remove specific knowledge or behaviors from a pre-trained language model using input-output pairs that represent the undesired information without retraining on the full dataset.
  • Figure 2: Overview of the LUNE framework. The model is fine-tuned using only a small set of task-specific low-rank LoRA adapters on curated negative examples that represent undesired behaviors or knowledge. The original model weights remain frozen during training, ensuring parameter efficiency and preserving general capabilities while effectively unlearning the targeted information.
  • Figure 3: Ablation study: negative vs. irrelevant examples. Fine-tuning with negative examples leads to higher unlearning effectiveness and robustness ($\uparrow$USR, $\uparrow$APR, $\downarrow$MIA), while maintaining high utility ($\uparrow$GUR), consistently outperforming irrelevant examples across all datasets.
  • Figure 4: The effect of low-rank $r$. Performance improves with larger $r$ up to 16, after which gains saturate. Moderate ranks (e.g., $r=8$ or $r=16$) offer the best trade-off.