PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications

TL;DR

PedCorpus is built, a high-quality dataset of over 300,000 multi-task instructions from pediatric textbooks, guidelines, and knowledge graph resources to fulfil diverse diagnostic demands and proposes PediatricsGPT, the first Chinese pediatric LLM assistant built on a systematic and robust training pipeline.

Abstract

Developing intelligent pediatric consultation systems offers promising prospects for improving diagnostic efficiency, especially in China, where healthcare resources are scarce. Despite recent advances in Large Language Models (LLMs) for Chinese medicine, their performance is sub-optimal in pediatric applications due to inadequate instruction data and vulnerable training procedures. To address the above issues, this paper builds PedCorpus, a high-quality dataset of over 300,000 multi-task instructions from pediatric textbooks, guidelines, and knowledge graph resources to fulfil diverse diagnostic demands. Upon well-designed PedCorpus, we propose PediatricsGPT, the first Chinese pediatric LLM assistant built on a systematic and robust training pipeline. In the continuous pre-training phase, we introduce a hybrid instruction pre-training mechanism to mitigate the internal-injected knowledge inconsistency of LLMs for medical domain adaptation. Immediately, the full-parameter Supervised Fine-Tuning (SFT) is utilized to incorporate the general medical knowledge schema into the models. After that, we devise a direct following preference optimization to enhance the generation of pediatrician-like humanistic responses. In the parameter-efficient secondary SFT phase, a mixture of universal-specific experts strategy is presented to resolve the competency conflict between medical generalist and pediatric expertise mastery. Extensive results based on the metrics, GPT-4, and doctor evaluations on distinct doctor downstream tasks show that PediatricsGPT consistently outperforms previous Chinese medical LLMs. Our model and dataset will be open-source for community development.

Figures (15)

• Figure 1: The sequential pipeline for developing PediatricsGPT. We begin by injecting intensive medical and world knowledge into the foundation model through the hybrid instruction mechanism in CPT phase. Then, full-parameter SFT is implemented to improve the model's instruction-following capabilities regarding medical generalists. After that, we introduce the direct following preference optimization to control the model behaviour to align with human preference. In the parameter-efficient SFT phase, the LoRA-based mixture of universal-specific experts is devised to mitigate conflicts across downstream tasks and competition between pediatric expertise and general mastery.
• Figure 2: Response comparisons of PediatricsGPT-13B with other baselines via GPT-4 evaluation.
• Figure 3: Response comparisons of PediatricsGPT-13B with other baselines via Doctor evaluation.
• Figure 4: Comparison results of different models on the CMD benchmark.
• Figure 5: Comparison results of different models on the webMedQA benchmark.