Hierarchical Dual-Strategy Unlearning for Biomedical and Healthcare Intelligence Using Imperfect and Privacy-Sensitive Medical Data

TL;DR

The paper tackles privacy and regulatory challenges in medical LLMs trained on imperfect data by proposing a hierarchical dual-strategy unlearning framework that simultaneously performs parameter-level geometric gradient projections and token-level concept-aware interventions. Guided by a unified four-level medical concept hierarchy, the method selectively forgets restricted surgical knowledge while preserving core medical reasoning, using a DP-augmented, LoRA-based fine-tuning approach. Empirical evaluation on MedMCQA and MHQA demonstrates strong forgetting (e.g., FR ≈ $82.7\%$) and preservation (≈ $88.5\%$) with minimal parameter updates (~0.1%), along with robust privacy (MIA resistant ≈ 0.89; AUC ≈ 0.555) and hierarchical concept preservation. The results indicate practical potential for privacy-compliant clinical AI, enabling targeted data removal, auditability, and efficient policy updates in real-world healthcare settings.

Abstract

Large language models (LLMs) exhibit exceptional performance but pose substantial privacy risks due to training data memorization, particularly within healthcare contexts involving imperfect or privacy-sensitive patient information. We present a hierarchical dual-strategy framework for selective knowledge unlearning that precisely removes specialized knowledge while preserving fundamental medical competencies. Our approach synergistically integrates geometric-constrained gradient updates to selectively modulate target parameters with concept-aware token-level interventions that distinguish between preservation-critical and unlearning-targeted tokens via a unified four-level medical concept hierarchy. Comprehensive evaluations on the MedMCQA (surgical) and MHQA (anxiety, depression, trauma) datasets demonstrate superior performance, achieving an 82.7% forgetting rate and 88.5% knowledge preservation. Notably, our framework maintains robust privacy guarantees while requiring modification of only 0.1% of parameters, addressing critical needs for regulatory compliance, auditability, and ethical standards in clinical research.

Figures (4)

• Figure 1: Architecture overview of the DuoLearn framework for medical knowledge unlearning. The system integrates medical data processing through embedding layers, concept-aware attention mechanisms for clinical concept retention and boundary management, DP-LoRA for privacy-preserving parameter updates, and comprehensive evaluation metrics (FR, KPR, MIA Score, HMTA) within a sequential training process that culminates in the MedForget deployment for clinical applications.
• Figure 2: Performance across different medical subdomains before and after unlearning. The surgical domain shows significant performance reduction after unlearning, while other medical domains maintain high performance levels, demonstrating selective unlearning effectiveness.
• Figure 3: Loss trajectories for different token categories during unlearning. Surgical tokens and memorized tokens show significant loss reduction, while general medical tokens maintain higher loss values, indicating selective preservation and demonstrating the effectiveness of our token-level analysis approach.
• Figure 4: Concept preservation performance across different knowledge hierarchy levels. Surgical knowledge shows consistent reduction across all levels, while general medical knowledge maintains high preservation rates, particularly at lower hierarchy levels, confirming the effectiveness of our hierarchical unlearning strategy.