Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LingLanMiDian: Systematic Evaluation of LLMs on TCM Knowledge and Clinical Reasoning

Rui Hua, Yu Wei, Zixin Shu, Kai Chang, Dengying Yan, Jianan Xia, Zeyu Liu, Hui Zhu, Shujie Song, Mingzhong Xiao, Xiaodong Li, Dongmei Jia, Zhuye Gao, Yanyan Meng, Naixuan Zhao, Yu Fu, Haibin Yu, Benman Yu, Yuanyuan Chen, Fei Dong, Zhizhou Meng, Pengcheng Yang, Songxue Zhao, Lijuan Pei, Yunhui Hu, Kan Ding, Jiayuan Duan, Wenmao Yin, Yang Gu, Runshun Zhang, Qiang Zhu, Jian Yu, Jiansheng Li, Baoyan Liu, Wenjia Wang, Xuezhong Zhou

TL;DR

LingLanMiDian presents a large-scale, expert-curated benchmark for evaluating LLMs on Traditional Chinese Medicine knowledge and clinical reasoning, unifying 13 subtasks across five domains with 44 evaluation dimensions and a 400-item hard subset per domain. It introduces a rigorous data-curation pipeline, synonym-tolerant and decision-recognition evaluation methods, and a macro-averaged metric framework that enables fair cross-model comparisons under zero-shot settings across 14 models. Key findings show near-ceiling recall on licensing-style knowledge tasks, but substantial gaps in information extraction from classical TCTextual sources and multi-step therapeutic reasoning, though reframing open-ended reasoning as constrained recognition improves reliability. The work highlights practical implications for domain-specific AI in medicine, underscores the need for robust, domain-aware evaluation, and paves the way for multimodal extensions and reproducible, clinically faithful TCMed AI research.

Abstract

Large language models (LLMs) are advancing rapidly in medical NLP, yet Traditional Chinese Medicine (TCM) with its distinctive ontology, terminology, and reasoning patterns requires domain-faithful evaluation. Existing TCM benchmarks are fragmented in coverage and scale and rely on non-unified or generation-heavy scoring that hinders fair comparison. We present the LingLanMiDian (LingLan) benchmark, a large-scale, expert-curated, multi-task suite that unifies evaluation across knowledge recall, multi-hop reasoning, information extraction, and real-world clinical decision-making. LingLan introduces a consistent metric design, a synonym-tolerant protocol for clinical labels, a per-dataset 400-item Hard subset, and a reframing of diagnosis and treatment recommendation into single-choice decision recognition. We conduct comprehensive, zero-shot evaluations on 14 leading open-source and proprietary LLMs, providing a unified perspective on their strengths and limitations in TCM commonsense knowledge understanding, reasoning, and clinical decision support; critically, the evaluation on Hard subset reveals a substantial gap between current models and human experts in TCM-specialized reasoning. By bridging fundamental knowledge and applied reasoning through standardized evaluation, LingLan establishes a unified, quantitative, and extensible foundation for advancing TCM LLMs and domain-specific medical AI research. All evaluation data and code are available at https://github.com/TCMAI-BJTU/LingLan and http://tcmnlp.com.

LingLanMiDian: Systematic Evaluation of LLMs on TCM Knowledge and Clinical Reasoning

TL;DR

LingLanMiDian presents a large-scale, expert-curated benchmark for evaluating LLMs on Traditional Chinese Medicine knowledge and clinical reasoning, unifying 13 subtasks across five domains with 44 evaluation dimensions and a 400-item hard subset per domain. It introduces a rigorous data-curation pipeline, synonym-tolerant and decision-recognition evaluation methods, and a macro-averaged metric framework that enables fair cross-model comparisons under zero-shot settings across 14 models. Key findings show near-ceiling recall on licensing-style knowledge tasks, but substantial gaps in information extraction from classical TCTextual sources and multi-step therapeutic reasoning, though reframing open-ended reasoning as constrained recognition improves reliability. The work highlights practical implications for domain-specific AI in medicine, underscores the need for robust, domain-aware evaluation, and paves the way for multimodal extensions and reproducible, clinically faithful TCMed AI research.

Abstract

Large language models (LLMs) are advancing rapidly in medical NLP, yet Traditional Chinese Medicine (TCM) with its distinctive ontology, terminology, and reasoning patterns requires domain-faithful evaluation. Existing TCM benchmarks are fragmented in coverage and scale and rely on non-unified or generation-heavy scoring that hinders fair comparison. We present the LingLanMiDian (LingLan) benchmark, a large-scale, expert-curated, multi-task suite that unifies evaluation across knowledge recall, multi-hop reasoning, information extraction, and real-world clinical decision-making. LingLan introduces a consistent metric design, a synonym-tolerant protocol for clinical labels, a per-dataset 400-item Hard subset, and a reframing of diagnosis and treatment recommendation into single-choice decision recognition. We conduct comprehensive, zero-shot evaluations on 14 leading open-source and proprietary LLMs, providing a unified perspective on their strengths and limitations in TCM commonsense knowledge understanding, reasoning, and clinical decision support; critically, the evaluation on Hard subset reveals a substantial gap between current models and human experts in TCM-specialized reasoning. By bridging fundamental knowledge and applied reasoning through standardized evaluation, LingLan establishes a unified, quantitative, and extensible foundation for advancing TCM LLMs and domain-specific medical AI research. All evaluation data and code are available at https://github.com/TCMAI-BJTU/LingLan and http://tcmnlp.com.
Paper Structure (42 sections, 9 equations, 6 figures, 4 tables)

This paper contains 42 sections, 9 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. General Domain Benchmarks
  4. Modern Biomedical Benchmarks
  5. Traditional Chinese Medicine Benchmarks
  6. LLMs for Chinese Medical and TCM Applications
  7. Methodology
  8. Overview
  9. Data Sources
  10. Data Curation and Quality Assurance
  11. Task Definition and Taxonomy
  12. Expert Annotation and Verification
  13. Knowledge-Oriented Tasks
  14. Information Extraction Tasks
  15. Diagnostic and Therapeutic Decision-Making Tasks
  16. ...and 27 more sections

Figures (6)

  • Figure 1: Overview of the LingLan construction pipeline. The left panel summarizes data sources; the center panel illustrates data processing and curation; the right panel presents the task taxonomy covered in LingLan together with the unified evaluation metrics. Abbreviations: HMC (Human-machine collaborate).
  • Figure 2: Subtask distribution and dataset sizes in LingLan. Abbreviations: FTK (Fundamental TCM Knowledge), CPMK (Chinese Patent Medicine Knowledge), TLE (TCM Licensing Examination), DR (Decision Recognition), IE (Information Extraction), DTR (Diagnostic–Therapeutic Reasoning), SC (single-choice).
  • Figure 3: Distribution of items across the 14 subjects in the TLE (TCM Licensing Examination) dataset.
  • Figure 4: Statistical analysis of entity distribution in IE tasks. (A) Top 10 most frequently mentioned entity types in modern medical records (left, blue) and ancient medical texts (right, orange). Vertical bars show absolute mention counts per category. (B) Distribution of entity mentions per sample for medical records (left, blue) and ancient texts (right, orange).
  • Figure 5: Model performance across all subtasks. Overall average refers to the mean performance across all tasks.
  • ...and 1 more figures