Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TCMD: A Traditional Chinese Medicine QA Dataset for Evaluating Large Language Models

Ping Yu, Kaitao Song, Fengchen He, Ming Chen, Jianfeng Lu

TL;DR

This work introduces TCMD, an exam-aligned MCQA dataset for Traditional Chinese Medicine derived from CNMLE, designed to objectively evaluate LLMs in the TCM domain. TCMD covers 5 realms and 18 subjects, balancing 2851 training and 600 test questions across four MCQA formats with explanations, and includes robustness analysis via answer-shuffling. Experimental results show general LLMs often outperform medical and TCM-specific models, while chain-of-thought prompts offer limited universal benefits in this domain; ensemble voting can improve robustness under certain conditions. The dataset and evaluation methodology provide a rigorous, objective benchmark to spur progress in LLMs for TCM and encourage careful consideration of reliability in real-world, risk-sensitive settings.

Abstract

The recently unprecedented advancements in Large Language Models (LLMs) have propelled the medical community by establishing advanced medical-domain models. However, due to the limited collection of medical datasets, there are only a few comprehensive benchmarks available to gauge progress in this area. In this paper, we introduce a new medical question-answering (QA) dataset that contains massive manual instruction for solving Traditional Chinese Medicine examination tasks, called TCMD. Specifically, our TCMD collects massive questions across diverse domains with their annotated medical subjects and thus supports us in comprehensively assessing the capability of LLMs in the TCM domain. Extensive evaluation of various general LLMs and medical-domain-specific LLMs is conducted. Moreover, we also analyze the robustness of current LLMs in solving TCM QA tasks by introducing randomness. The inconsistency of the experimental results also reveals the shortcomings of current LLMs in solving QA tasks. We also expect that our dataset can further facilitate the development of LLMs in the TCM area.

TCMD: A Traditional Chinese Medicine QA Dataset for Evaluating Large Language Models

TL;DR

This work introduces TCMD, an exam-aligned MCQA dataset for Traditional Chinese Medicine derived from CNMLE, designed to objectively evaluate LLMs in the TCM domain. TCMD covers 5 realms and 18 subjects, balancing 2851 training and 600 test questions across four MCQA formats with explanations, and includes robustness analysis via answer-shuffling. Experimental results show general LLMs often outperform medical and TCM-specific models, while chain-of-thought prompts offer limited universal benefits in this domain; ensemble voting can improve robustness under certain conditions. The dataset and evaluation methodology provide a rigorous, objective benchmark to spur progress in LLMs for TCM and encourage careful consideration of reliability in real-world, risk-sensitive settings.

Abstract

The recently unprecedented advancements in Large Language Models (LLMs) have propelled the medical community by establishing advanced medical-domain models. However, due to the limited collection of medical datasets, there are only a few comprehensive benchmarks available to gauge progress in this area. In this paper, we introduce a new medical question-answering (QA) dataset that contains massive manual instruction for solving Traditional Chinese Medicine examination tasks, called TCMD. Specifically, our TCMD collects massive questions across diverse domains with their annotated medical subjects and thus supports us in comprehensively assessing the capability of LLMs in the TCM domain. Extensive evaluation of various general LLMs and medical-domain-specific LLMs is conducted. Moreover, we also analyze the robustness of current LLMs in solving TCM QA tasks by introducing randomness. The inconsistency of the experimental results also reveals the shortcomings of current LLMs in solving QA tasks. We also expect that our dataset can further facilitate the development of LLMs in the TCM area.
Paper Structure (31 sections, 1 equation, 2 figures, 4 tables)

This paper contains 31 sections, 1 equation, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs for Medical Application
  4. Medical Evaluation Datasets for LLMs
  5. Dataset Construction
  6. Manual Instruction of CNMLE
  7. Question Types
  8. Exam Content
  9. Data Collecting and Processing
  10. Problems Collection
  11. Manual Verification and Processing
  12. Balance of Dataset
  13. Data Statistics
  14. Experiments and Analysis
  15. Experimental Setup
  16. ...and 16 more sections

Figures (2)

  • Figure 1: The detailed exam content of the CNMLE for TCM. The mark * indicates that the subject is not required for those with apprenticeships or demonstrable expertise.
  • Figure 2: The accuracy of LLMs under different thresholds of consistency. The dashed line is the mean accuracy of the 120 variants from Table \ref{['tab:shuffle_distribution']}.