Auditing Language Model Unlearning via Information Decomposition

TL;DR

The paper addresses privacy and regulatory concerns in LLMs by showing that existing unlearning methods can leave persistent, information-bearing traces of forgotten data in internal representations. It introduces an interpretable information-theoretic audit based on Partial Information Decomposition (PID) to separate unlearned (unique to the base) from residual knowledge (shared across base and unlearned models) and demonstrates that residual information correlates with vulnerability to adversarial reconstruction. Using Redundant Information Neural Estimation (RINE) and Blackwell sufficiency concepts, the authors quantify unlearned and residual information and validate the approach on TOFU and MUSE benchmarks across multiple models, revealing that gradient-based methods often retain substantial residual knowledge while representation- or preference-based methods better reduce it. They further propose an inference-time abstention mechanism, risk-scoring samples based on base-unlearned probe agreement, to mitigate privacy leakage without severely sacrificing utility. Overall, the work provides a principled, white-box auditing framework that yields actionable tools for safer deployment of LLMs and regulatory compliance.

Abstract

We expose a critical limitation in current approaches to machine unlearning in language models: despite the apparent success of unlearning algorithms, information about the forgotten data remains linearly decodable from internal representations. To systematically assess this discrepancy, we introduce an interpretable, information-theoretic framework for auditing unlearning using Partial Information Decomposition (PID). By comparing model representations before and after unlearning, we decompose the mutual information with the forgotten data into distinct components, formalizing the notions of unlearned and residual knowledge. Our analysis reveals that redundant information, shared across both models, constitutes residual knowledge that persists post-unlearning and correlates with susceptibility to known adversarial reconstruction attacks. Leveraging these insights, we propose a representation-based risk score that can guide abstention on sensitive inputs at inference time, providing a practical mechanism to mitigate privacy leakage. Our work introduces a principled, representation-level audit for unlearning, offering theoretical insight and actionable tools for safer deployment of language models.

Figures (3)

• Figure 1: Existing unlearning methods may appear effective but retain forgotten data internally (A). Our framework (B) uses Partial Information Decomposition to reveal residual knowledge, which (C) correlates with attack risk and informs a privacy-aware abstention score.
• Figure 2: Probe AUROC scores on the TOFU benchmark for different unlearning algorithms on llama. Forget set information is linearly decodable from the residual stream of LLMs, thus undermining the sense of privacy provided by the apparent success of surface-level metrics like Forget Quality.
• Figure 3: Partial Information Decomposition. The box denotes the total joint mutual information $I_{}(Y;B,U)$, which is decomposed into four non-negative terms: synergistic information $I_\text{syn}$, redundant information $\color{Dandelion}{I_\cap}$ and the two unique information terms $\color{SkyBlue}{I^B_{\text{uniq}}}$ and $I_\text{uniq}^U$.

Theorems & Definitions (3)

• Definition 5.1: Unlearned Knowledge
• Definition 5.2: Residual Knowledge
• Definition 5.3: Redundant information