Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Representation-Aware Unlearning via Activation Signatures: From Suppression to Knowledge-Signature Erasure

Syed Naveed Mahmood, Md. Rezaur Rahman Bhuiyan, Tasfia Zaman, Jareen Tasneem Khondaker, Md. Sameer Sakib, Nazia Tasnim, Farig Sadeque

TL;DR

The paper tackles the challenge of true knowledge erasure in LLMs by moving beyond surface output suppression to a representation-aware approach that targets internal activation signatures. It introduces the Knowledge Immunization Framework (KIF), a three-stage pipeline that localizes subject-specific activation directions, applies lightweight Knowledge Suppression Capsules, and distills suppression into a global LoRA adapter via a Self-Healing Loop. Across standard and reasoning-prior models (3B–14B), KIF achieves near-oracle forgetting with minimal utility loss on standard architectures and reveals capacity-dependent erasure behavior in reasoning models, validated by a dual-metric evaluation that separates obfuscation from true erasure. This mechanistic approach breaks the traditional stability-erasure trade-off and offers a scalable, PEFT-based solution with significant privacy and safety implications for deploying LLMs. The work highlights the importance of internal representations for durable unlearning and lays groundwork for future mechanistic objectives that explicitly model knowledge persistence across model families and scales.

Abstract

Selective knowledge erasure from LLMs is critical for GDPR compliance and model safety, yet current unlearning methods conflate behavioral suppression with true knowledge removal, allowing latent capabilities to persist beneath surface-level refusals. In this work, we address this challenge by introducing Knowledge Immunization Framework (KIF), a representation-aware architecture that distinguishes genuine erasure from obfuscation by targeting internal activation signatures rather than surface outputs. Our approach combines dynamic suppression of subject-specific representations with parameter-efficient adaptation, enabling durable unlearning without full model retraining. KIF achieves near-oracle erasure (FQ approx 0.99 vs. 1.00) while preserving utility at oracle levels (MU = 0.62), effectively breaking the stability-erasure tradeoff that has constrained all prior work. We evaluate both standard foundation models (Llama and Mistral) and reasoning-prior models (Qwen and DeepSeek) across 3B to 14B parameters. Our observation shows that standard models exhibit scale-independent true erasure (<3% utility drift), while reasoning-prior models reveal fundamental architectural divergence. Our comprehensive dual-metric evaluation protocol, combining surface-level leakage with latent trace persistence, operationalizes the obfuscation - erasure distinction and enables the first systematic diagnosis of mechanism-level forgetting behavior across model families and scales.

Representation-Aware Unlearning via Activation Signatures: From Suppression to Knowledge-Signature Erasure

TL;DR

The paper tackles the challenge of true knowledge erasure in LLMs by moving beyond surface output suppression to a representation-aware approach that targets internal activation signatures. It introduces the Knowledge Immunization Framework (KIF), a three-stage pipeline that localizes subject-specific activation directions, applies lightweight Knowledge Suppression Capsules, and distills suppression into a global LoRA adapter via a Self-Healing Loop. Across standard and reasoning-prior models (3B–14B), KIF achieves near-oracle forgetting with minimal utility loss on standard architectures and reveals capacity-dependent erasure behavior in reasoning models, validated by a dual-metric evaluation that separates obfuscation from true erasure. This mechanistic approach breaks the traditional stability-erasure trade-off and offers a scalable, PEFT-based solution with significant privacy and safety implications for deploying LLMs. The work highlights the importance of internal representations for durable unlearning and lays groundwork for future mechanistic objectives that explicitly model knowledge persistence across model families and scales.

Abstract

Selective knowledge erasure from LLMs is critical for GDPR compliance and model safety, yet current unlearning methods conflate behavioral suppression with true knowledge removal, allowing latent capabilities to persist beneath surface-level refusals. In this work, we address this challenge by introducing Knowledge Immunization Framework (KIF), a representation-aware architecture that distinguishes genuine erasure from obfuscation by targeting internal activation signatures rather than surface outputs. Our approach combines dynamic suppression of subject-specific representations with parameter-efficient adaptation, enabling durable unlearning without full model retraining. KIF achieves near-oracle erasure (FQ approx 0.99 vs. 1.00) while preserving utility at oracle levels (MU = 0.62), effectively breaking the stability-erasure tradeoff that has constrained all prior work. We evaluate both standard foundation models (Llama and Mistral) and reasoning-prior models (Qwen and DeepSeek) across 3B to 14B parameters. Our observation shows that standard models exhibit scale-independent true erasure (<3% utility drift), while reasoning-prior models reveal fundamental architectural divergence. Our comprehensive dual-metric evaluation protocol, combining surface-level leakage with latent trace persistence, operationalizes the obfuscation - erasure distinction and enables the first systematic diagnosis of mechanism-level forgetting behavior across model families and scales.
Paper Structure (36 sections, 6 equations, 4 figures, 12 tables)

This paper contains 36 sections, 6 equations, 4 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Dataset Curation
  5. Activation-Signature Extraction
  6. Capsule Forging
  7. Geometric Suppression Operator
  8. Self-Healing Loop
  9. Experiments and Results
  10. Model and Baseline Selection
  11. Evaluation Metrics
  12. Real-World Entity Dataset Results
  13. General Capability Retention
  14. TOFU Benchmark Results
  15. Result Analysis and Discussion
  16. ...and 21 more sections

Figures (4)

  • Figure 1: Comparison of approaches: while standard methods may yield obfuscation that hides answers without removing latent knowledge, KIF targets the internal signature to achieve true erasure.
  • Figure 2: Knowledge Immunization Framework (KIF) pipeline. Subject-driven prompts are used to probe MLP-layer activations, from which subject-specific activation signatures are mined. These signatures instantiate lightweight suppression capsules attached to high-salience MLP layers for immediate suppression. A self-healing loop distills this behavior into a global LoRA adapter, yielding durable parameter-level unlearning where targeted knowledge is completely removed from the model weight
  • Figure 3: EL10 ratio (log scale; left axis) and surface leakage (SMR; right axis). Standard models maintain low internal activation (EL10 $<1$) regardless of size, whereas reasoning models display a distinct capacity-dependent U-curve.
  • Figure 4: Distribution of extracted knowledge triples per subject. The dataset exhibits class imbalance, with Beyoncé (18.0%) and Taylor Swift (15.7%) having the highest representation, compared to minority classes like Queen (2.0%).