Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Democratizing Multilingual Large Language Models For Medicine Through A Two-Stage Instruction Fine-tuning Approach

Meng Zhou, Surajsinh Parmar, Anubhav Bhatti

TL;DR

This work introduces two multilingual instruction fine-tuning datasets, MMed-IFT and MMed-IFT-MC, containing over 200k high-quality medical samples in six languages and proposes a two-stage training paradigm, which achieves competitive results on both English and multilingual benchmarks.

Abstract

Open-source, multilingual medical large language models (LLMs) have the potential to serve linguistically diverse populations across different regions. Adapting generic LLMs for healthcare often requires continual pretraining, but this approach is computationally expensive and sometimes impractical. Instruction fine-tuning on a specific task may not always guarantee optimal performance due to the lack of broader domain knowledge that the model needs to understand and reason effectively in diverse scenarios. To address these challenges, we introduce two multilingual instruction fine-tuning datasets, MMed-IFT and MMed-IFT-MC, containing over 200k high-quality medical samples in six languages. We propose a two-stage training paradigm: the first stage injects general medical knowledge using MMed-IFT, while the second stage fine-tunes task-specific multiple-choice questions with MMed-IFT-MC. Our method achieves competitive results on both English and multilingual benchmarks, striking a balance between computational efficiency and performance. We plan to make our dataset and model weights public at \url{https://github.com/SpassMed/Med-Llama3} in the future.

Towards Democratizing Multilingual Large Language Models For Medicine Through A Two-Stage Instruction Fine-tuning Approach

TL;DR

This work introduces two multilingual instruction fine-tuning datasets, MMed-IFT and MMed-IFT-MC, containing over 200k high-quality medical samples in six languages and proposes a two-stage training paradigm, which achieves competitive results on both English and multilingual benchmarks.

Abstract

Open-source, multilingual medical large language models (LLMs) have the potential to serve linguistically diverse populations across different regions. Adapting generic LLMs for healthcare often requires continual pretraining, but this approach is computationally expensive and sometimes impractical. Instruction fine-tuning on a specific task may not always guarantee optimal performance due to the lack of broader domain knowledge that the model needs to understand and reason effectively in diverse scenarios. To address these challenges, we introduce two multilingual instruction fine-tuning datasets, MMed-IFT and MMed-IFT-MC, containing over 200k high-quality medical samples in six languages. We propose a two-stage training paradigm: the first stage injects general medical knowledge using MMed-IFT, while the second stage fine-tunes task-specific multiple-choice questions with MMed-IFT-MC. Our method achieves competitive results on both English and multilingual benchmarks, striking a balance between computational efficiency and performance. We plan to make our dataset and model weights public at \url{https://github.com/SpassMed/Med-Llama3} in the future.
Paper Structure (7 sections, 2 equations, 2 figures, 2 tables)

This paper contains 7 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. MMed-IFT and MMed-IFT-MC Dataset
  4. Two-Stage IFT Approach
  5. Experiments
  6. Results
  7. Conclusion

Figures (2)

  • Figure 1: An overview of this work. Our MMed-IFT dataset is based on various GPT-augmented open-resourced datasets and our hand-crafted samples. The MMed-IFT-MC dataset is based on public MMedBench and KorMedMCQA datasets with GPT-generated rationale. MMed-IFT dataset contains a wide range of medical questions presented in the question-answer format, which is used in Stage 1. MMed-IFT-MC is a more task-specific dataset tailored for Medical Licenses Examination-style multiple-choice questions, which is used in Stage 2.
  • Figure 2: An example of preprocessing methods we utilized in this work. From left to right: answer expanding, question-answer pair generation, and cross-lingual translation.