MedForget: Hierarchy-Aware Multimodal Unlearning Testbed for Medical AI

TL;DR

MedForget introduces a hierarchy-aware multimodal unlearning benchmark for medical AI, organizing data into Institution→Patient→Study→Section and evaluating forget/retain performance across multiple granularities. Using MIMIC-CXR as the data backbone, the study tests four unlearning methods and demonstrates clear trade-offs between forgetting strength and diagnostic utility, with coarse-grained forgetting delivering stronger privacy at the expense of performance and fine-grained forgetting preserving utility but risking leakage. The paper also presents a hierarchical reconstruction attack to assess leakage risk and highlights the need for structure-aware unlearning to meet regulatory demands under HIPAA and GDPR. Overall, MedForget provides a practical, HIPAA-aligned evaluation framework to guide development of safe, compliant medical AI systems that can forget sensitive information without catastrophically degrading clinical reasoning.

Abstract

Pretrained Multimodal Large Language Models (MLLMs) are increasingly deployed in medical AI systems for clinical reasoning, diagnosis support, and report generation. However, their training on sensitive patient data raises critical privacy and compliance challenges under regulations such as HIPAA and GDPR, which enforce the "right to be forgotten". Unlearning, the process of tuning models to selectively remove the influence of specific training data points, offers a potential solution, yet its effectiveness in complex medical settings remains underexplored. To systematically study this, we introduce MedForget, a Hierarchy-Aware Multimodal Unlearning Testbed with explicit retain and forget splits and evaluation sets containing rephrased variants. MedForget models hospital data as a nested hierarchy (Institution -> Patient -> Study -> Section), enabling fine-grained assessment across eight organizational levels. The benchmark contains 3840 multimodal (image, question, answer) instances, each hierarchy level having a dedicated unlearning target, reflecting distinct unlearning challenges. Experiments with four SOTA unlearning methods on three tasks (generation, classification, cloze) show that existing methods struggle to achieve complete, hierarchy-aware forgetting without reducing diagnostic performance. To test whether unlearning truly deletes hierarchical pathways, we introduce a reconstruction attack that progressively adds hierarchical level context to prompts. Models unlearned at a coarse granularity show strong resistance, while fine-grained unlearning leaves models vulnerable to such reconstruction. MedForget provides a practical, HIPAA-aligned testbed for building compliant medical AI systems.

Figures (8)

• Figure 1: MedForget enables hierarchical multimodal unlearning. Unlike existing unlearning benchmarks that adopt a flat structure, treating all data points as independent and unconnected, MedForget introduces a clinically inspired hierarchical organization (Institution$\rightarrow$Patient$\rightarrow$Study$\rightarrow$Section). This structure mirrors the organization of real-world medical data and supports the systematic evaluation of selective unlearning across multiple granularities. Models can thus be fine-tuned and unlearned at varying hierarchy levels, enabling analysis of how forgetting propagates across semantically and structurally related entities.
• Figure 2: Hierarchical benchmark design and unlearning setup. Illustration of the forget–retain partition at each hierarchy level, where approximately 25% of entities (red) form the forget set and the remainder constitute the retain set. This setup supports controlled multi-level unlearning experiments, capturing how forgetting propagates across hierarchically related data.
• Figure 3: Examples of data subsets in MedForget. Each subset serves a distinct evaluation purpose: the Forget Set (a) contains target information to be unlearned, the Forget Rephrase Set (c) tests generalization through paraphrased questions and augmented views, the Retain Set (b) evaluates preservation of medical knowledge that should not be forgotten, and the General Med Set (d) assesses retention of general medical capabilities on an independent benchmark. All examples show medical images paired with questions and ground truth answers tailored to their evaluation objectives. Additional data examples are provided in Appendix \ref{['fig:dataset-examples-appendix']}.
• Figure 4: Distribution of section types in MedForget.
• Figure 5: Performance comparison of unlearning methods on the Forget Set, Retain Set, Forget Rephrase Set, and General Med Set across different hierarchy levels. The x-axis labels denote the following hierarchy levels: Inst. (Institute), Pat. (Patient), Stu. (Study), and Sec. (Section). The Forget Rephrase and General Med sets are evaluation subsets.