Towards Training A Chinese Large Language Model for Anesthesiology

TL;DR

This work presents Hypnos, a Chinese anesthesia-focused LLM built by extending Llama with Chinese vocabulary and a general-to-specific training regime. It introduces a cross-filtering data curation pipeline to improve data quality and a two-stage training process that first leverages broad general medicine data before specializing on anesthesia data. A standardized benchmark comprising AneQA, AneCQ, and AneRC is developed to evaluate anesthesia-specific capabilities, with Hypnos outperforming multiple medical LLM baselines in automatic, manual, and GPT-4-based assessments. The study demonstrates that combining high-quality real-world anesthesia data with carefully filtered model-generated data and domain-aware training yields a competitive, potentially clinically useful anesthesia LLM. This work advances field-specific LLM development and provides a practical framework for building robust domain-focused medical AI tools.

Abstract

Medical large language models (LLMs) have gained popularity recently due to their significant practical utility. However, most existing research focuses on general medicine, and there is a need for in-depth study of LLMs in specific fields like anesthesiology. To fill the gap, we introduce Hypnos, a Chinese Anesthesia model built upon existing LLMs, e.g., Llama. Hypnos' contributions have three aspects: 1) The data, such as utilizing Self-Instruct, acquired from current LLMs likely includes inaccuracies. Hypnos implements a cross-filtering strategy to improve the data quality. This strategy involves using one LLM to assess the quality of the generated data from another LLM and filtering out the data with low quality. 2) Hypnos employs a general-to-specific training strategy that starts by fine-tuning LLMs using the general medicine data and subsequently improving the fine-tuned LLMs using data specifically from Anesthesiology. The general medical data supplement the medical expertise in Anesthesiology and enhance the effectiveness of Hypnos' generation. 3) We introduce a standardized benchmark for evaluating medical LLM in Anesthesiology. Our benchmark includes both publicly available instances from the Internet and privately obtained cases from the Hospital. Hypnos outperforms other medical LLMs in anesthesiology in metrics, GPT-4, and human evaluation on the benchmark dataset.

Figures (4)

• Figure 1: The brief training process of our Hypnos. The training process contains two aspects: data collection and model training. In the data collection, we provide the data types, sizes, and sources. In the model training processes, Hypnos first extends the Chinese Vocabulary, then tunes the Llama with LORA using the general medical data, and last, fully refines the model using anesthesia instruction data. The Lock refers to the component that is fixed and not learnable.
• Figure 2: Step 1: Use GPT-3.5-turbo and Claude to generate anesthesia-related instruction data. Step 2: Clean up data with obvious incorrectness. Step 3: Use Claude to score the data generated by GPT-3.5-turbo, and use GPT-3.5-turbo to score the data generated by Claude. Step 4: Clean up instruction data with low scores to obtain the final model generation data.
• Figure 3: Human Assessment Results of Hypnos and other models on the AneRC and AneQA.
• Figure 4: Manual evaluation variance for each indicator on the dataset AneRC and AneQA.