SemEval-2025 Task 4: Unlearning sensitive content from Large Language Models

TL;DR

SemEval-2025 Task 4 presents a multitask benchmark to study unlearning sensitive content from LLMs across three domains: synthetic creative documents, synthetic PII biographies, and real biographies from Wikipedia. The framework combines regurgitation and knowledge tests with forget and retain data splits, two fixed candidate models, and a set of evaluation metrics including ROUGE-L, exact-match QA, MIA, and MMLU utility, culminating in an aggregate final score. Results from over 100 submissions show that gradient based unlearning with LoRA adapters and selective parameter updates yields strong forgetting while largely preserving utility, yet trade-offs between leakage risk and performance remain. The findings highlight challenges and propose directions for future metrics, larger models, and broader unlearning targets.

Abstract

We introduce SemEval-2025 Task 4: unlearning sensitive content from Large Language Models (LLMs). The task features 3 subtasks for LLM unlearning spanning different use cases: (1) unlearn long form synthetic creative documents spanning different genres; (2) unlearn short form synthetic biographies containing personally identifiable information (PII), including fake names, phone number, SSN, email and home addresses, and (3) unlearn real documents sampled from the target model's training dataset. We received over 100 submissions from over 30 institutions and we summarize the key techniques and lessons in this paper.

Figures (7)

• Figure 1: Performance on retain and forget subsets for benchmarked unlearning algorithms. Reg: Regurgitation Rate ($r$), Kno: Knowledge Accuracy ($t$). Split refers to data subset (forget or retain) used in evaluations.
• Figure 2: Distribution of key scores for all participants on both models. MMLU plots are zoomed in (but still contain 10 bins). Dashed line indicates threshold for 7B model utility below which submissions are discarded.
• Figure 3: Distribution of participant scores for forget and retain sets on the 7B model for all 6 sub-tasks.
• Figure 4: Distribution of participant scores for forget and retain sets on the 1B model for all 6 sub-tasks.
• Figure 5: Examples of full documents and test prompts for the three tasks covered in this challenge. The figure is quoted from ramakrishna2025lumellmunlearningmultitask.