TCM-Ladder: A Benchmark for Multimodal Question Answering on Traditional Chinese Medicine

TL;DR

TCM-Ladder presents the first large-scale, multimodal QA benchmark for Traditional Chinese Medicine, spanning fundamental theory, diagnostics, formulas, pharmacognosy, and clinical subfields, with text, image, audio, and video modalities. It introduces Ladder-Score, a domain-specific metric combining terminological accuracy and semantic quality to better evaluate TCM QA. The authors benchmark nine general-domain and five TCM-specific LLMs, fine-tune two models (BenCao and Ladder-base), and demonstrate that domain-focused training yields substantial gains on structured TCM tasks. An accompanying open platform (tcmladder.com) provides data, leaderboard access, and ongoing updates to support reproducibility and community-driven expansion. The work highlights the importance of multimodal fusion and domain-specific evaluation for advancing LLMs in specialized medical domains and sets a foundation for fair, scalable benchmarking in TCM AI research.

Abstract

Traditional Chinese Medicine (TCM), as an effective alternative medicine, has been receiving increasing attention. In recent years, the rapid development of large language models (LLMs) tailored for TCM has highlighted the urgent need for an objective and comprehensive evaluation framework to assess their performance on real-world tasks. However, existing evaluation datasets are limited in scope and primarily text-based, lacking a unified and standardized multimodal question-answering (QA) benchmark. To address this issue, we introduce TCM-Ladder, the first comprehensive multimodal QA dataset specifically designed for evaluating large TCM language models. The dataset covers multiple core disciplines of TCM, including fundamental theory, diagnostics, herbal formulas, internal medicine, surgery, pharmacognosy, and pediatrics. In addition to textual content, TCM-Ladder incorporates various modalities such as images and videos. The dataset was constructed using a combination of automated and manual filtering processes and comprises over 52,000 questions. These questions include single-choice, multiple-choice, fill-in-the-blank, diagnostic dialogue, and visual comprehension tasks. We trained a reasoning model on TCM-Ladder and conducted comparative experiments against nine state-of-the-art general domain and five leading TCM-specific LLMs to evaluate their performance on the dataset. Moreover, we propose Ladder-Score, an evaluation method specifically designed for TCM question answering that effectively assesses answer quality in terms of terminology usage and semantic expression. To the best of our knowledge, this is the first work to systematically evaluate mainstream general domain and TCM-specific LLMs on a unified multimodal benchmark. The datasets and leaderboard are publicly available at https://tcmladder.com and will be continuously updated.

Figures (4)

• Figure 1: Overview of the architectural composition of TCM-Ladder. TCM-Ladder encompasses six task types aimed at evaluating the comprehensive capabilities of large language models in Traditional Chinese Medicine. These include: (1) single-choice questions, which assess basic knowledge recognition; (2) multiple-choice questions, designed to test the model’s ability to integrate and reason over complex concepts; (3) long-form diagnostic question answering, which evaluates clinical reasoning based on detailed symptom descriptions and patient inquiries; (4) fill-in-the-blank tasks, which measure generative accuracy and contextual understanding without the aid of answer options; (5) image-based comprehension tasks, involving the interpretation of medicinal herb and tongue images to assess multimodal reasoning across visual and textual inputs; and (6) additional audio and video resources, such as diagnostic sounds, pulse recordings, and tuina (massage) videos, which support the development and evaluation of multimodal TCM models incorporating auditory and dynamic visual data.
• Figure 2: Data distribution and length statistics in TCM-Ladder. The left illustrates the dataset composition across text, image, and audio modalities, along with TCM subfields. The right plots show the distribution of question and answer lengths.
• Figure 3: Performance of general-domain and TCM-specific language models on single and multiple-choice question answering tasks.
• Figure 4: The performance of large language models on questions regarding Chinese herbal medicine and tongue image classification.