Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parameter-Efficient Token Embedding Editing for Clinical Class-Level Unlearning

Iyad Ait Hou, Shrenik Borad, Harsh Sharma, Pooja Srinivasan, Rebecca Hwa, Aya Zirikly

Abstract

Machine unlearning is increasingly important for clinical language models, where privacy regulations and institutional policies may require removing sensitive information from deployed systems without retraining from scratch. In practice, deletion requests must balance effective forgetting of targeted information with preservation of model utility and minimal parameter modification. We introduce Sparse Token Embedding Unlearning (STEU), a parameter-efficient method for behavioral class-level unlearning that updates only PMI-selected token embeddings together with a small classifier head while keeping all encoder layers frozen. Across experiments on MIMIC-IV, MIMIC-III, and eICU using BioClinicalBERT, BERT-base, and DistilBERT, STEU consistently suppresses the target class while largely preserving retained task performance. In the primary MIMIC-IV setting, STEU achieves near-complete forgetting (forget F1 = 0.0004) while maintaining competitive retained utility (retain avg F1 = 0.4766) after modifying only 0.19\% of model parameters. These results suggest that targeted behavioral unlearning can be achieved through sparse embedding edits without modifying deeper encoder representations.

Parameter-Efficient Token Embedding Editing for Clinical Class-Level Unlearning

Abstract

Machine unlearning is increasingly important for clinical language models, where privacy regulations and institutional policies may require removing sensitive information from deployed systems without retraining from scratch. In practice, deletion requests must balance effective forgetting of targeted information with preservation of model utility and minimal parameter modification. We introduce Sparse Token Embedding Unlearning (STEU), a parameter-efficient method for behavioral class-level unlearning that updates only PMI-selected token embeddings together with a small classifier head while keeping all encoder layers frozen. Across experiments on MIMIC-IV, MIMIC-III, and eICU using BioClinicalBERT, BERT-base, and DistilBERT, STEU consistently suppresses the target class while largely preserving retained task performance. In the primary MIMIC-IV setting, STEU achieves near-complete forgetting (forget F1 = 0.0004) while maintaining competitive retained utility (retain avg F1 = 0.4766) after modifying only 0.19\% of model parameters. These results suggest that targeted behavioral unlearning can be achieved through sparse embedding edits without modifying deeper encoder representations.
Paper Structure (7 sections, 4 equations, 3 figures, 5 tables)

This paper contains 7 sections, 4 equations, 3 figures, 5 tables.

Table of Contents

  1. Abstract
  2. Introduction
  3. Related Work
  4. Methodology
  5. Experimental Setup
  6. Results
  7. Conclusion

Figures (3)

  • Figure 1: Conceptual overview of Sparse Token Embedding Unlearning (STEU). While existing machine unlearning methods modify large portions of encoder parameters, STEU performs localized edits to selected token embeddings, enabling class-level behavioral unlearning with minimal parameter updates.
  • Figure 2: STEU workflow figure combining token selection and constrained unlearning. The pipeline first identifies class-discriminative tokens via frequency-weighted PMI, then updates only the selected embedding rows and classifier head with gradient masking while all encoder layers remain frozen.
  • Figure 3: Utility–parameter tradeoff. Each point shows retained F1 versus number of updated parameters (log scale), with forget F1 annotated. STEU occupies the high-utility, low-parameter region.