Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TCMBench: A Comprehensive Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine

Wenjing Yue, Xiaoling Wang, Wei Zhu, Ming Guan, Huanran Zheng, Pengfei Wang, Changzhi Sun, Xin Ma

TL;DR

TCMBench introduces a dedicated benchmark for evaluating large language models in Traditional Chinese Medicine by combining the TCM-ED question dataset with TMNLI for semantic entailment evaluation and a domain-specific TCMScore metric that fuses term-matching with semantic consistency. The benchmark reveals that current LLMs underperform in TCM tasks, though larger general models (e.g., GPT-4) show the strongest performance, and domain knowledge can improve results with caveats about potential degradation in core reasoning when heavily fine-tuned. TCMBench also demonstrates that traditional generation metrics (Rouge, BertScore) can mislead due to text-length effects, whereas TCMScore provides a more faithful assessment of TCM knowledge accuracy and semantic alignment. The work emphasizes the need to balance pretraining with domain specificity and outlines directions for reducing hallucinations and expanding data to include clinical decision processes such as syndrome differentiation and treatment planning, thereby advancing practical LLM use in TCM.

Abstract

Large language models (LLMs) have performed remarkably well in various natural language processing tasks by benchmarking, including in the Western medical domain. However, the professional evaluation benchmarks for LLMs have yet to be covered in the traditional Chinese medicine(TCM) domain, which has a profound history and vast influence. To address this research gap, we introduce TCM-Bench, an comprehensive benchmark for evaluating LLM performance in TCM. It comprises the TCM-ED dataset, consisting of 5,473 questions sourced from the TCM Licensing Exam (TCMLE), including 1,300 questions with authoritative analysis. It covers the core components of TCMLE, including TCM basis and clinical practice. To evaluate LLMs beyond accuracy of question answering, we propose TCMScore, a metric tailored for evaluating the quality of answers generated by LLMs for TCM related questions. It comprehensively considers the consistency of TCM semantics and knowledge. After conducting comprehensive experimental analyses from diverse perspectives, we can obtain the following findings: (1) The unsatisfactory performance of LLMs on this benchmark underscores their significant room for improvement in TCM. (2) Introducing domain knowledge can enhance LLMs' performance. However, for in-domain models like ZhongJing-TCM, the quality of generated analysis text has decreased, and we hypothesize that their fine-tuning process affects the basic LLM capabilities. (3) Traditional metrics for text generation quality like Rouge and BertScore are susceptible to text length and surface semantic ambiguity, while domain-specific metrics such as TCMScore can further supplement and explain their evaluation results. These findings highlight the capabilities and limitations of LLMs in the TCM and aim to provide a more profound assistance to medical research.

TCMBench: A Comprehensive Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine

TL;DR

TCMBench introduces a dedicated benchmark for evaluating large language models in Traditional Chinese Medicine by combining the TCM-ED question dataset with TMNLI for semantic entailment evaluation and a domain-specific TCMScore metric that fuses term-matching with semantic consistency. The benchmark reveals that current LLMs underperform in TCM tasks, though larger general models (e.g., GPT-4) show the strongest performance, and domain knowledge can improve results with caveats about potential degradation in core reasoning when heavily fine-tuned. TCMBench also demonstrates that traditional generation metrics (Rouge, BertScore) can mislead due to text-length effects, whereas TCMScore provides a more faithful assessment of TCM knowledge accuracy and semantic alignment. The work emphasizes the need to balance pretraining with domain specificity and outlines directions for reducing hallucinations and expanding data to include clinical decision processes such as syndrome differentiation and treatment planning, thereby advancing practical LLM use in TCM.

Abstract

Large language models (LLMs) have performed remarkably well in various natural language processing tasks by benchmarking, including in the Western medical domain. However, the professional evaluation benchmarks for LLMs have yet to be covered in the traditional Chinese medicine(TCM) domain, which has a profound history and vast influence. To address this research gap, we introduce TCM-Bench, an comprehensive benchmark for evaluating LLM performance in TCM. It comprises the TCM-ED dataset, consisting of 5,473 questions sourced from the TCM Licensing Exam (TCMLE), including 1,300 questions with authoritative analysis. It covers the core components of TCMLE, including TCM basis and clinical practice. To evaluate LLMs beyond accuracy of question answering, we propose TCMScore, a metric tailored for evaluating the quality of answers generated by LLMs for TCM related questions. It comprehensively considers the consistency of TCM semantics and knowledge. After conducting comprehensive experimental analyses from diverse perspectives, we can obtain the following findings: (1) The unsatisfactory performance of LLMs on this benchmark underscores their significant room for improvement in TCM. (2) Introducing domain knowledge can enhance LLMs' performance. However, for in-domain models like ZhongJing-TCM, the quality of generated analysis text has decreased, and we hypothesize that their fine-tuning process affects the basic LLM capabilities. (3) Traditional metrics for text generation quality like Rouge and BertScore are susceptible to text length and surface semantic ambiguity, while domain-specific metrics such as TCMScore can further supplement and explain their evaluation results. These findings highlight the capabilities and limitations of LLMs in the TCM and aim to provide a more profound assistance to medical research.
Paper Structure (18 sections, 1 equation, 15 figures, 6 tables, 1 algorithm)

This paper contains 18 sections, 1 equation, 15 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Proposed Benchmark
  4. Overview
  5. Construction and Statistics
  6. TCM-ED
  7. TMNLI
  8. Evaluation
  9. Models
  10. Experimental Settings
  11. Evaluation Metrics
  12. Main Results
  13. Analysis of LLMs Accuracy.
  14. Analysis of expression quality of LLMs.
  15. Human evaluation
  16. ...and 3 more sections

Figures (15)

  • Figure 1: The difference between TCM and Western Medicine.
  • Figure 2: The performance of different LLMs on different branches of TCM-Bench.
  • Figure 3: The overview of TCM-Bench. It consists of two parts: (1) On the left is the construction process of the TCM-ED dataset. (2) On the right is the evaluation process of TCM-Bench. The bottom section showcases the TMNLI dataset and the TCM-Deberta model, as well as the RGB]235,240,255TCMScore metric.
  • Figure 4: Branches of TCM in TCM-ED.
  • Figure 5: The total accuracy results on category of TCMLE.
  • ...and 10 more figures