Development and bilingual evaluation of Japanese medical large language model within reasonably low computational resources

TL;DR

This work tackles the practicality gap of medical LLMs by building a 7B parameter model JMedLLM-v1 through a two stage training pipeline MFPT and MPEFT to achieve strong bilingual performance in Japanese and English. A unified evaluation framework using IgakuQA, MedQA, MedMCQA, MMLU and JMMLU benchmarks demonstrates that the 7B model can match or exceed many 70B open source models on several tasks while operating under low resource constraints. The study reveals cross lingual knowledge transfer and latent knowledge extraction through fine tuning, with Japanese data yielding improvements in both languages. The authors provide publicly available evaluation code and discuss practical implications for deploying medical LLMs in clinical settings with limited computational resources.

Abstract

The recent success of large language models (LLMs) and the scaling law has led to a widespread adoption of larger models. Particularly in the healthcare industry, there is an increasing demand for locally operated LLMs due to security concerns. However, the majority of high quality open-source LLMs have a size of 70B parameters, imposing significant financial burdens on users for GPU preparation and operation. To overcome these issues, we present a medical adaptation based on the recent 7B models, which enables the operation in low computational resources. We compare the performance on medical question-answering benchmarks in two languages (Japanese and English), demonstrating that its scores reach parity with or surpass those of currently existing medical LLMs that are ten times larger. We find that fine-tuning an English-centric base model on Japanese medical dataset improves the score in both language, supporting the effect of cross-lingual knowledge transfer. We hope that this study will alleviate financial challenges, serving as a stepping stone for clinical institutions to practically utilize LLMs locally. Our evaluation code is available at https://github.com/stardust-coder/japanese-lm-med-harness.