Harry Potter is Still Here! Probing Knowledge Leakage in Targeted Unlearned Large Language Models via Automated Adversarial Prompting

TL;DR

The paper introduces LURK, an automated adversarial prompting framework to detect latent retained knowledge in unlearned LLMs, specifically targeting Harry Potter content. By optimizing adversarial suffixes with a calibrated knowledge-checking module, LURK reveals leakage that standard unlearning benchmarks may miss, showing that some methods conceal rather than forget, especially in larger models. Across multiple models and unlearning baselines, findings indicate a trade-off between forgetting efficacy and utility, with leakage increasing under probing despite apparent forgetting in some settings. The work emphasizes the need for rigorous, verifiable unlearning evaluation and suggests broader applicability beyond the Harry Potter domain, accompanied by code release for transparency and reproducibility.

Abstract

This work presents LURK (Latent UnleaRned Knowledge), a novel framework that probes for hidden retained knowledge in unlearned LLMs through adversarial suffix prompting. LURK automatically generates adversarial prompt suffixes designed to elicit residual knowledge about the Harry Potter domain, a commonly used benchmark for unlearning. Our experiments reveal that even models deemed successfully unlearned can leak idiosyncratic information under targeted adversarial conditions, highlighting critical limitations of current unlearning evaluation standards. By uncovering latent knowledge through indirect probing, LURK offers a more rigorous and diagnostic tool for assessing the robustness of unlearning algorithms. All code will be publicly available.

Figures (6)

• Figure 1: LURK generates adversarial prompt suffix to prob knowledge leakage in unlearned LLMs.
• Figure 2: Overall process of LURK in generating adversarial suffix tokens $x_\mathcal{A}$.
• Figure A1: Distributions of number of leakage, correct Harry Potter references in the generated texts of LLMs under knowledge leakage probing via LURK (outliers above 20 are removed for clarify).
• Figure A2: Validation prompts with step-by-step instructions (Chain-of-Thought)
• Figure A3: Validation prompt with step-by-step instructions and ground truth scoring examples (Chain-of-Thought + Few-shot)