Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Erase to Retain: Low Rank Adaptation Guided Selective Unlearning in Medical Segmentation Networks

Nirjhor Datta, Md. Golam Rabiul Alam

TL;DR

This work tackles the challenge of selective unlearning in dense medical segmentation by proposing Erase to Retain, a lightweight framework that freezes the backbone and trains low-rank LoRA adapters to erase lesion-specific information on a designated forget subset $\\mathcal{D}_f$ while preserving general segmentation ability on the retain set $\\mathcal{D}_r$. It combines a retain-preserving objective (supervised loss, knowledge distillation, and guard terms) with a forgetting objective (background-focused segmentation and entropy/random-label strategies), all confined to the LoRA subspace to limit model drift. Theoretical analysis positions unlearning as constrained optimization within a subspace with bounded parameter drift, and empirical results on ISIC 2018 and CHASE_DB1 show substantial forgetting on the forget set with minimal degradation on retain and validation performance, including cross-domain and cross-task generalization to classification. The method offers an efficient, reversible pathway toward privacy-preserving medical imaging models, enabling targeted erasure of sensitive information without full retraining. Overall, Erase to Retain demonstrates that LoRA-based subspace unlearning can achieve precise, controllable forgetting in dense prediction tasks with practical implications for clinical deployment and regulatory compliance.

Abstract

The ability to selectively remove knowledge from medical segmentation networks is increasingly important for privacy compliance, ethical deployment, and continual dataset revision. We introduce Erase to Retain, a controllable unlearning framework for medical image segmentation that achieves targeted forgetting without full retraining. Our method uses a teacher-student distillation paradigm with Low-Rank Adaptation (LoRA) constrained subspace updates, enabling the student network to erase lesion-specific or class-specific representations in low-rank decoder spaces while preserving global anatomical understanding. During the strong unlearning phase, LoRA modules are adversarially optimized to contradict the teacher's confident predictions on a designated forget subset, enforcing semantic removal. This is followed by a gentle restoration phase that recovers generalization on retained data through head-only supervised refinement. For ISIC segmentation, the student reduces forget-set IoU from 0.875 to 0.509 while maintaining competitive performance on the retain and validation splits (0.647 to 0.677 IoU). On the cross-domain CHASE dataset, Erase to Retain consistently lowers forget-set IoU while preserving utility on retain and validation sets. For ISIC classification, our method decreases accuracy on the forget subset from 87.0 percent to 64.1 percent while improving retain accuracy from 83.9 percent to 90.6 percent. These results demonstrate that LoRA-based subspace unlearning provides a practical pathway toward responsible, controllable, and reversible unlearning in medical image analysis, enabling models to forget sensitive samples or structures while preserving performance where it matters most.

Erase to Retain: Low Rank Adaptation Guided Selective Unlearning in Medical Segmentation Networks

TL;DR

This work tackles the challenge of selective unlearning in dense medical segmentation by proposing Erase to Retain, a lightweight framework that freezes the backbone and trains low-rank LoRA adapters to erase lesion-specific information on a designated forget subset while preserving general segmentation ability on the retain set . It combines a retain-preserving objective (supervised loss, knowledge distillation, and guard terms) with a forgetting objective (background-focused segmentation and entropy/random-label strategies), all confined to the LoRA subspace to limit model drift. Theoretical analysis positions unlearning as constrained optimization within a subspace with bounded parameter drift, and empirical results on ISIC 2018 and CHASE_DB1 show substantial forgetting on the forget set with minimal degradation on retain and validation performance, including cross-domain and cross-task generalization to classification. The method offers an efficient, reversible pathway toward privacy-preserving medical imaging models, enabling targeted erasure of sensitive information without full retraining. Overall, Erase to Retain demonstrates that LoRA-based subspace unlearning can achieve precise, controllable forgetting in dense prediction tasks with practical implications for clinical deployment and regulatory compliance.

Abstract

The ability to selectively remove knowledge from medical segmentation networks is increasingly important for privacy compliance, ethical deployment, and continual dataset revision. We introduce Erase to Retain, a controllable unlearning framework for medical image segmentation that achieves targeted forgetting without full retraining. Our method uses a teacher-student distillation paradigm with Low-Rank Adaptation (LoRA) constrained subspace updates, enabling the student network to erase lesion-specific or class-specific representations in low-rank decoder spaces while preserving global anatomical understanding. During the strong unlearning phase, LoRA modules are adversarially optimized to contradict the teacher's confident predictions on a designated forget subset, enforcing semantic removal. This is followed by a gentle restoration phase that recovers generalization on retained data through head-only supervised refinement. For ISIC segmentation, the student reduces forget-set IoU from 0.875 to 0.509 while maintaining competitive performance on the retain and validation splits (0.647 to 0.677 IoU). On the cross-domain CHASE dataset, Erase to Retain consistently lowers forget-set IoU while preserving utility on retain and validation sets. For ISIC classification, our method decreases accuracy on the forget subset from 87.0 percent to 64.1 percent while improving retain accuracy from 83.9 percent to 90.6 percent. These results demonstrate that LoRA-based subspace unlearning provides a practical pathway toward responsible, controllable, and reversible unlearning in medical image analysis, enabling models to forget sensitive samples or structures while preserving performance where it matters most.

Paper Structure

This paper contains 34 sections, 18 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Machine Unlearning
  4. Unlearning in Large Language Models
  5. LoRA and Parameter-Efficient Adaptation
  6. Medical Image Segmentation and Privacy
  7. Theoretical Analysis
  8. Method
  9. Problem Formulation
  10. LoRA Adapter Parameterization
  11. Retain-Preserving Objective
  12. 1. Supervised Loss.
  13. 2. Knowledge Distillation.
  14. 3. Guard Loss.
  15. Forgetting Objective
  16. ...and 19 more sections

Figures (3)

  • Figure 1: Overview of our proposed unlearning methodology. The pipeline consists of: (1) teacher model inference, (2) LoRA-based student adaptation, (3) retain-guided knowledge distillation, and (4) background-focused forgetting.
  • Figure 2: Qualitative analysis of unlearning on ISIC. Rows 1–3 show samples from the retain set, while rows 4–6 show samples from the forget set. For each image we visualize the input, ground-truth lesion mask (red), teacher prediction (blue), student prediction after unlearning (green), and the absolute difference $|T-S|$ between teacher and student masks. On retain samples, teacher and student predictions closely match and foreground probabilities inside the lesion region remain high (T_fg $\approx$ S_fg). On forget samples, the student explicitly suppresses lesion evidence (T_fg $\gg$ S_fg) and the difference map is concentrated on the forgotten lesion area, demonstrating targeted forgetting without collateral damage.
  • Figure 3: Classwise F1 comparison between Teacher and LoRA-based Student across Retain, Forget, and Test splits on the ISIC 9-class dataset. The Student retains high performance on Retain classes, shows deliberate degradation on Forget classes (successful unlearning), and maintains competitive generalization on Test classes.