Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation

Yusheng Liao, Shuyang Jiang, Zhe Chen, Yanfeng Wang, Yu Wang

TL;DR

A Medical LLM through decoupling Clinical Alignment and Knowledge Aggregation (MedCare), which is designed to achieve state-of-the-art (SOTA) performance on over 20 medical tasks, as well as SOTA results on specific medical alignment tasks.

Abstract

Large language models (LLMs) have shown substantial progress in natural language understanding and generation, proving valuable especially in the medical field. Despite advancements, challenges persist due to the complexity and diversity inherent in medical tasks, which can be categorized as knowledge-intensive tasks and alignment-required tasks. Previous approaches either ignore the latter task or focus on a minority of tasks and hence lose generalization. To address these drawbacks, we propose a progressive fine-tuning pipeline. This pipeline employs a Knowledge Aggregator and a Noise aggregator to encode diverse knowledge in the first stage and filter out detrimental information. In the second stage, we drop the Noise Aggregator to avoid the interference of suboptimal representation and leverage an additional alignment module optimized towards an orthogonal direction to the knowledge space to mitigate knowledge forgetting. Based on this two-stage paradigm, we proposed a Medical LLM through decoupling Clinical Alignment and Knowledge Aggregation (MedCare), which is designed to achieve state-of-the-art (SOTA) performance on over 20 medical tasks, as well as SOTA results on specific medical alignment tasks. Various model sizes of MedCare (1.8B, 7B, 14B) all demonstrate significant improvements over existing models with similar model sizes.

MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation

TL;DR

A Medical LLM through decoupling Clinical Alignment and Knowledge Aggregation (MedCare), which is designed to achieve state-of-the-art (SOTA) performance on over 20 medical tasks, as well as SOTA results on specific medical alignment tasks.

Abstract

Large language models (LLMs) have shown substantial progress in natural language understanding and generation, proving valuable especially in the medical field. Despite advancements, challenges persist due to the complexity and diversity inherent in medical tasks, which can be categorized as knowledge-intensive tasks and alignment-required tasks. Previous approaches either ignore the latter task or focus on a minority of tasks and hence lose generalization. To address these drawbacks, we propose a progressive fine-tuning pipeline. This pipeline employs a Knowledge Aggregator and a Noise aggregator to encode diverse knowledge in the first stage and filter out detrimental information. In the second stage, we drop the Noise Aggregator to avoid the interference of suboptimal representation and leverage an additional alignment module optimized towards an orthogonal direction to the knowledge space to mitigate knowledge forgetting. Based on this two-stage paradigm, we proposed a Medical LLM through decoupling Clinical Alignment and Knowledge Aggregation (MedCare), which is designed to achieve state-of-the-art (SOTA) performance on over 20 medical tasks, as well as SOTA results on specific medical alignment tasks. Various model sizes of MedCare (1.8B, 7B, 14B) all demonstrate significant improvements over existing models with similar model sizes.
Paper Structure (39 sections, 10 equations, 11 figures, 7 tables)

This paper contains 39 sections, 10 equations, 11 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Low-rank Adaptation (LoRA)
  4. Mixture of LoRA
  5. MedCare
  6. Problem Formulation
  7. Miscellaneous Knowledge Aggregation
  8. Downstream Alignment
  9. Experiments
  10. Results on Knowledge-Intensive Tasks
  11. Results on Alignment-Required Tasks
  12. Ablation Experiments
  13. Discussion
  14. Response to RQ1: Noise Aggregator interferes with models because of the routing mismatch.
  15. Response to RQ2: Yes. Symmetric structure failed to decouple the learning process into the common knowledge and task-specific requirements.
  16. ...and 24 more sections

Figures (11)

  • Figure 1: Examples of two types of medical tasks. Knowledge-intensive tasks require models to possess sufficient knowledge, whereas alignment-required tasks additionally necessitate the model to meet specific requirements criteria.
  • Figure 2: Overview of the proposed MedCare. In the MKA stage, MedCare encodes advantageous knowledge and noisy contents with Knowledge Aggregator and Noise Aggregator from both types of tasks, respectively. The updated Noise Aggregator is removed to avoid the knowledge disruption. In the DA stage, an additional alignment module and orthogonal regularization are introduced to cater to the requirements of the alignment tasks.
  • Figure 3: Results on 16 tasks in CBLUE. ChatGLM-6B is fine-tuned on the CBLUE dataset with LoRA and ChatGPT is augmented by in-context learning. The results are obtained from the official implementation of CBLUE.
  • Figure 4: Mismatch between expert activation times and performance on CEval. (a) Activation times of each expert combination. (b) Performance of only used each expert combination. The $i$-th raw and $j$-th column indicates the combinations of the $i$-th and $j$-th experts. The accuracy of the vanilla MoLoRA inference is 45.00.
  • Figure 5: Performance with different sizes of alignment-required datasets for DA fine-tuning stage. (a) The average performance on knowledge-intensive tasks. (b) The average performance on CBLUE. (c) The average performance on CCTE. 'MedCarew/o MKA' indicates the model is fine-tuned with only the second stage. Note that the score of the knowledge examinations is the average of the CMMLU, CEval, PLE Pharamy, and PLE TCM.
  • ...and 6 more figures