LoRKD: Low-Rank Knowledge Decomposition for Medical Foundation Models

TL;DR

A novel framework named Low-Rank Knowledge Decomposition (LoRKD), which explicitly separates gradients from different tasks by incorporating low-rank expert modules and efficient knowledge separation convolution, and significantly improves algorithm efficiency by achieving knowledge separation within a single forward propagation.

Abstract

The widespread adoption of large-scale pre-training techniques has significantly advanced the development of medical foundation models, enabling them to serve as versatile tools across a broad range of medical tasks. However, despite their strong generalization capabilities, medical foundation models pre-trained on large-scale datasets tend to suffer from domain gaps between heterogeneous data, leading to suboptimal performance on specific tasks compared to specialist models, as evidenced by previous studies. In this paper, we explore a new perspective called "Knowledge Decomposition" to improve the performance on specific medical tasks, which deconstructs the foundation model into multiple lightweight expert models, each dedicated to a particular anatomical region, with the aim of enhancing specialization and simultaneously reducing resource consumption. To accomplish the above objective, we propose a novel framework named Low-Rank Knowledge Decomposition (LoRKD), which explicitly separates gradients from different tasks by incorporating low-rank expert modules and efficient knowledge separation convolution. The low-rank expert modules resolve gradient conflicts between heterogeneous data from different anatomical regions, providing strong specialization at lower costs. The efficient knowledge separation convolution significantly improves algorithm efficiency by achieving knowledge separation within a single forward propagation. Extensive experimental results on segmentation and classification tasks demonstrate that our decomposed models not only achieve state-of-the-art performance but also exhibit superior transferability on downstream tasks, even surpassing the original foundation models in task-specific evaluations. The code is available at here.

Figures (10)

• Figure 1: Knowledge decomposition is employed to break down the foundation model into multiple lightweight expert models, each tailored to a specific domain. The goal of this paradigm is to improve the specialization of deployment models within a specific domain, while simultaneously reducing deployment costs.
• Figure 2: The resource consumption of foundation models is growing at an exponential rate. The size of the circle represents the model's parameters.
• Figure 3: Performance comparison between the foundation model and specialist model. $\triangle$ DSC is the DSC value of nnUNet minus the DSC value of MedSAM.
• Figure 4: The illustration of LoRKD for medical foundation models on segmentation. The low-rank expert modules control the number of parameters and efficient knowledge separation convolution (EKS Conv) achieves computationally efficient gradient separation. Decomposed models can replace medical foundation model in specific domains and can switch task knowledge conveniently between departments. The case for classification tasks holds by turning the decoders as classifiers.
• Figure 5: Data distribution in two large-scale medical datasets.