RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models

TL;DR

The authors address the challenge of erasing real-world, memorized knowledge from large language models by introducing RWKU, a zero-shot unlearning benchmark built around 200 famous individuals. RWKU employs a multi-faceted evaluation framework encompassing forget and retain assessments, including membership inference attacks and a broad set of adversarial probes to stress forgotten knowledge and neighboring content. Through experiments with two models and six baselines, the study reveals limitations in current methods, notably in batch-target unlearning and robustness to MIAs, while showing that synthetic forget content can aid forgetting but may not fully preserve truthfulness and factuality. The work provides a valuable, publicly available benchmark for advancing practical LLM unlearning, with clear directions for expanding knowledge sources and evaluation rigor.

Abstract

Large language models (LLMs) inevitably memorize sensitive, copyrighted, and harmful knowledge from the training corpus; therefore, it is crucial to erase this knowledge from the models. Machine unlearning is a promising solution for efficiently removing specific knowledge by post hoc modifying models. In this paper, we propose a Real-World Knowledge Unlearning benchmark (RWKU) for LLM unlearning. RWKU is designed based on the following three key factors: (1) For the task setting, we consider a more practical and challenging unlearning setting, where neither the forget corpus nor the retain corpus is accessible. (2) For the knowledge source, we choose 200 real-world famous people as the unlearning targets and show that such popular knowledge is widely present in various LLMs. (3) For the evaluation framework, we design the forget set and the retain set to evaluate the model's capabilities across various real-world applications. Regarding the forget set, we provide four four membership inference attack (MIA) methods and nine kinds of adversarial attack probes to rigorously test unlearning efficacy. Regarding the retain set, we assess locality and utility in terms of neighbor perturbation, general ability, reasoning ability, truthfulness, factuality, and fluency. We conduct extensive experiments across two unlearning scenarios, two models and six baseline methods and obtain some meaningful findings. We release our benchmark and code publicly at http://rwku-bench.github.io for future work.

Figures (6)

• Figure 1: Memorization quantification of different knowledge sources.
• Figure 2: The evaluation framework of RWKU.
• Figure 3: Trade off between unlearning efficacy, locality and model utility of LLaMA3-Instruct (8B).
• Figure 4: Comparison of different adversarial attack types on LLaMA3-Instruct (8B).
• Figure 5: Results of batch-target unleaning experiments on LLaMA3-Instruct (8B).