Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DUET: Distilled LLM Unlearning from an Efficiently Contextualized Teacher

Yisheng Zhong, Zhengbang Yang, Zhuangdi Zhu

TL;DR

This work tackles the problem of unlearning undesirable knowledge in LLMs without full retraining by introducing DUET, a distillation-based framework that transfers in-context refusal behavior from a prompt-conditioned teacher into a student model via Top-K logit distillation. By aligning the student’s top logits with the teacher’s responses on forgotten content, while incorporating retention data, DUET achieves strong forgetting with minimal degradation to general utility. Empirical results on HP-related content (MUSE) and WMDP benchmarks show DUET outperforms existing methods in balancing forgetting and utility, exhibiting robustness to reverse prompting and across different evaluation formats, all with data-efficient training. The approach also includes expanded evaluation protocols and demonstrates generalizability across multiple LLMs, offering a scalable path toward trustworthy unlearning in real-world settings.

Abstract

LLM unlearning is a technique to remove the impacts of undesirable knowledge from the model without retraining from scratch, which is indispensable towards trustworthy AI. Existing unlearning methods face significant limitations: conventional tuning-based unlearning is computationally heavy and prone to catastrophic forgetting. In contrast, in-contextualized unlearning is lightweight for precise unlearning but vulnerable to prompt removal or reverse engineering attacks. In response, we propose Distilled Unlearning from an Efficient Teacher (DUET), a novel distillation-based unlearning method that combines the merits of these two lines of work. It learns a student model to imitate the behavior of a prompt-steered teacher that effectively refuses undesirable knowledge generation while preserving general domain knowledge. Extensive evaluations on existing benchmarks with our enriched evaluation protocols demonstrate that DUET achieves higher performance in both forgetting and utility preservation, while being orders of magnitude more data-efficient than state-of-the-art unlearning methods.

DUET: Distilled LLM Unlearning from an Efficiently Contextualized Teacher

TL;DR

This work tackles the problem of unlearning undesirable knowledge in LLMs without full retraining by introducing DUET, a distillation-based framework that transfers in-context refusal behavior from a prompt-conditioned teacher into a student model via Top-K logit distillation. By aligning the student’s top logits with the teacher’s responses on forgotten content, while incorporating retention data, DUET achieves strong forgetting with minimal degradation to general utility. Empirical results on HP-related content (MUSE) and WMDP benchmarks show DUET outperforms existing methods in balancing forgetting and utility, exhibiting robustness to reverse prompting and across different evaluation formats, all with data-efficient training. The approach also includes expanded evaluation protocols and demonstrates generalizability across multiple LLMs, offering a scalable path toward trustworthy unlearning in real-world settings.

Abstract

LLM unlearning is a technique to remove the impacts of undesirable knowledge from the model without retraining from scratch, which is indispensable towards trustworthy AI. Existing unlearning methods face significant limitations: conventional tuning-based unlearning is computationally heavy and prone to catastrophic forgetting. In contrast, in-contextualized unlearning is lightweight for precise unlearning but vulnerable to prompt removal or reverse engineering attacks. In response, we propose Distilled Unlearning from an Efficient Teacher (DUET), a novel distillation-based unlearning method that combines the merits of these two lines of work. It learns a student model to imitate the behavior of a prompt-steered teacher that effectively refuses undesirable knowledge generation while preserving general domain knowledge. Extensive evaluations on existing benchmarks with our enriched evaluation protocols demonstrate that DUET achieves higher performance in both forgetting and utility preservation, while being orders of magnitude more data-efficient than state-of-the-art unlearning methods.
Paper Structure (38 sections, 4 equations, 2 figures, 18 tables, 1 algorithm)

This paper contains 38 sections, 4 equations, 2 figures, 18 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Preliminary of Training-Based LLM Unlearning
  5. In-Context Unlearning Provides Efficient Supervision Signal
  6. A Unified Unlearning Objective for Top-K Logit Distillation
  7. Experiments
  8. Datasets Preparation.
  9. Baseline Methods
  10. Metrics.
  11. Performance Overview
  12. Effects of Logit Distillation:
  13. Impacts of In-Context Unlearning Prompts
  14. Unlearning Robustness Against Reverse Engineering
  15. Unlearning Robustness Against Evaluation Format Variation
  16. ...and 23 more sections

Figures (2)

  • Figure 1: Top-10 logits for a Harry Potter related query before and after DUET unlearning. Multi-token words are shown complete for clarity. Before unlearning, domain-related and affirmative tokens dominate. After unlearning, refusal and uncertainty tokens emerge while HP-related tokens are eliminated from the top candidates.
  • Figure 2: Comparing DUET with conventional unlearning that requires sequentially supervised unlearning on each response token.