Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMs

Clément Christophe, Tathagata Raha, Svetlana Maslenkova, Muhammad Umar Salman, Praveen K Kanithi, Marco AF Pimentel, Shadab Khan

TL;DR

While continuous pretraining beyond 250 billion tokens yields marginal improvements on its own, it establishes a strong foundation for instruct fine-tuning, and NEFTune, designed primarily to enhance generation quality, surprisingly demonstrates additional gains on the authors' benchmark.

Abstract

Large Language Models (LLMs) have demonstrated significant potential in transforming clinical applications. In this study, we investigate the efficacy of four techniques in adapting LLMs for clinical use-cases: continuous pretraining, instruct fine-tuning, NEFTune, and prompt engineering. We employ these methods on Mistral 7B and Mixtral 8x7B models, leveraging a large-scale clinical pretraining dataset of 50 billion tokens and an instruct fine-tuning dataset of 500 million tokens. Our evaluation across various clinical tasks reveals the impact of each technique. While continuous pretraining beyond 250 billion tokens yields marginal improvements on its own, it establishes a strong foundation for instruct fine-tuning. Notably, NEFTune, designed primarily to enhance generation quality, surprisingly demonstrates additional gains on our benchmark. Complex prompt engineering methods further enhance performance. These findings show the importance of tailoring fine-tuning strategies and exploring innovative techniques to optimize LLM performance in the clinical domain.

Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMs

TL;DR

While continuous pretraining beyond 250 billion tokens yields marginal improvements on its own, it establishes a strong foundation for instruct fine-tuning, and NEFTune, designed primarily to enhance generation quality, surprisingly demonstrates additional gains on the authors' benchmark.

Abstract

Large Language Models (LLMs) have demonstrated significant potential in transforming clinical applications. In this study, we investigate the efficacy of four techniques in adapting LLMs for clinical use-cases: continuous pretraining, instruct fine-tuning, NEFTune, and prompt engineering. We employ these methods on Mistral 7B and Mixtral 8x7B models, leveraging a large-scale clinical pretraining dataset of 50 billion tokens and an instruct fine-tuning dataset of 500 million tokens. Our evaluation across various clinical tasks reveals the impact of each technique. While continuous pretraining beyond 250 billion tokens yields marginal improvements on its own, it establishes a strong foundation for instruct fine-tuning. Notably, NEFTune, designed primarily to enhance generation quality, surprisingly demonstrates additional gains on our benchmark. Complex prompt engineering methods further enhance performance. These findings show the importance of tailoring fine-tuning strategies and exploring innovative techniques to optimize LLM performance in the clinical domain.
Paper Structure (23 sections, 1 equation, 5 figures, 3 tables)

This paper contains 23 sections, 1 equation, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Experiments
  4. Continuous Pretraining
  5. Instruct Fine-Tuning
  6. NEFTune
  7. Prompt Engineering
  8. Hardware infrastructure.
  9. Datasets
  10. Pretraining Dataset
  11. Finetuning Dataset
  12. Evaluations
  13. Results
  14. Non-instructed models can't be evaluated on QA tasks.
  15. Instruct Fine-tuning Specializes the Model for QA Data
  16. ...and 8 more sections

Figures (5)

  • Figure 1: Training loss for Mixtral during continuous pretraining on the general (left) and clinical (right) subsets.
  • Figure 2: Evolution of MedQA accuracy for Mistral-7b and Mixtral 8x7b base models as well as our instructed versions of Mistral-7b during continuous pretraining. $Pˆt$: Continuous Pretrained with variable numbers of tokens $t$, $F$: Instruct Finetuned. We show that, while base model accuracy remains consistent, applying instruct-finetuning leads to notable improvements.
  • Figure 3: Evolution of MedQA accuracy using MedPrompt over different versions of Mixtral.
  • Figure 4: Zero-shot prompt format on a sample from MedQA
  • Figure 5: Chain-of-Thought prompt format on a sample from MedQA