Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

70B-parameter large language models in Japanese medical question-answering

Issey Sukeda, Risa Kishikawa, Satoshi Kodera

TL;DR

This work utilizes multiple 70B-parameter LLMs for the first time and shows that instruction tuning using Japanese medical question-answering dataset significantly improves the ability of Japanese LLMs to solve Japanese medical license exams, surpassing 50\% in accuracy.

Abstract

Since the rise of large language models (LLMs), the domain adaptation has been one of the hot topics in various domains. Many medical LLMs trained with English medical dataset have made public recently. However, Japanese LLMs in medical domain still lack its research. Here we utilize multiple 70B-parameter LLMs for the first time and show that instruction tuning using Japanese medical question-answering dataset significantly improves the ability of Japanese LLMs to solve Japanese medical license exams, surpassing 50\% in accuracy. In particular, the Japanese-centric models exhibit a more significant leap in improvement through instruction tuning compared to their English-centric counterparts. This underscores the importance of continual pretraining and the adjustment of the tokenizer in our local language. We also examine two slightly different prompt formats, resulting in non-negligible performance improvement.

70B-parameter large language models in Japanese medical question-answering

TL;DR

This work utilizes multiple 70B-parameter LLMs for the first time and shows that instruction tuning using Japanese medical question-answering dataset significantly improves the ability of Japanese LLMs to solve Japanese medical license exams, surpassing 50\% in accuracy.

Abstract

Since the rise of large language models (LLMs), the domain adaptation has been one of the hot topics in various domains. Many medical LLMs trained with English medical dataset have made public recently. However, Japanese LLMs in medical domain still lack its research. Here we utilize multiple 70B-parameter LLMs for the first time and show that instruction tuning using Japanese medical question-answering dataset significantly improves the ability of Japanese LLMs to solve Japanese medical license exams, surpassing 50\% in accuracy. In particular, the Japanese-centric models exhibit a more significant leap in improvement through instruction tuning compared to their English-centric counterparts. This underscores the importance of continual pretraining and the adjustment of the tokenizer in our local language. We also examine two slightly different prompt formats, resulting in non-negligible performance improvement.
Paper Structure (23 sections, 3 figures, 5 tables, 1 algorithm)

This paper contains 23 sections, 3 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Medical Instruction Tuning in Japanese
  3. Base Model
  4. QLoRA
  5. Instruction Dataset
  6. Evaluation
  7. Evaluation Dataset
  8. Metrics
  9. Results
  10. Base Model Selection : Swallow outperforms Xwin
  11. Format of CoT Prompts
  12. Comparison with GPT-4
  13. Limitations and Future Works
  14. Conclusion
  15. QLoRA Hyperparameters
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Overview of our candidate LLMs
  • Figure 2: Improvement by QLoRA instruction tuning in Accuracy. Gray shows the performance of Llama 2 as baseline. Light blue shows the difference between Xwin (original) and Llama 2 (original). Pink shows the difference between Swallow (original) and Llama 2 (original), which is negative in #3-2(A). Blue shows the contribution of QLoRA.
  • Figure : Evaluation of the correctness for each question-answer pair