From Beginner to Expert: Modeling Medical Knowledge into General LLMs

TL;DR

The paper tackles the challenge of enabling reliable medical reasoning in LLMs by proposing a 3-stage optimization that starts from a general pre-trained LLM (AntGLM-10B) and progressively injects medical knowledge, tunes domain-specific instructions, and adapts to specific medical tasks. It introduces comprehensive datasets for each stage and a novel Verification-of-Choice prompting method to boost multi-choice reasoning. The resulting AntGLM-Med-10B achieves competitive PubMedQA performance (80.6%), close to larger models, and demonstrates that careful architecture, data, and prompting design can compensate for smaller model sizes in medical contexts. This work provides a practical pathway for building physician-oriented LLMs with strong domain fidelity and task adaptability at modest model scales.

Abstract

Recently, large language model (LLM) based artificial intelligence (AI) systems have demonstrated remarkable capabilities in natural language understanding and generation. However, these models face a significant challenge when it comes to sensitive applications, such as reasoning over medical knowledge and answering medical questions in a physician-like manner. Prior studies attempted to overcome this challenge by increasing the model size (>100B) to learn more general medical knowledge, while there is still room for improvement in LLMs with smaller-scale model sizes (<100B). In this work, we start from a pre-trained general LLM model (AntGLM-10B) and fine-tune it from a medical beginner towards a medical expert (called AntGLM-Med-10B), which leverages a 3-stage optimization procedure, i.e., general medical knowledge injection, medical domain instruction tuning, and specific medical task adaptation. Our contributions are threefold: (1) We specifically investigate how to adapt a pre-trained general LLM in medical domain, especially for a specific medical task. (2) We collect and construct large-scale medical datasets for each stage of the optimization process. These datasets encompass various data types and tasks, such as question-answering, medical reasoning, multi-choice questions, and medical conversations. (3) Specifically for multi-choice questions in the medical domain, we propose a novel Verification-of-Choice approach for prompting engineering, which significantly enhances the reasoning ability of LLMs. Remarkably, by combining the above approaches, our AntGLM-Med-10B model can outperform the most of LLMs on PubMedQA, including both general and medical LLMs, even when these LLMs have larger model size.

Figures (6)

• Figure 1: The 3-stage optimization procedure of AntGLM-Med-10B. We collect different data types (e.g., medical articles, books, and examinations) and medical tasks (question answering, reasoning, and multi-choice questions). In the adaptation stage, we mainly consider the optimizations for multi-choice questions, resulting in competitive performance on the PubMedQA.
• Figure 2: The detailed techniques for different optimization stages.
• Figure 3: A comparison example for Chain-of-Thought and Verification-of-Choice.
• Figure 4: Overview of $\texttt{C-Poly}$ framework.
• Figure 5: The PubMedQA learderboard.