Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MLB: A Scenario-Driven Benchmark for Evaluating Large Language Models in Clinical Applications

Qing He, Dongsheng Bi, Jianrong Lu, Minghui Yang, Zixiao Chen, Jiacheng Lu, Jing Chen, Nannan Du, Xiao Cu, Sijing Wu, Peng Xiang, Yinyin Hu, Yi Guo, Chunpu Li, Shaoyang Li, Zhuo Dong, Ming Jiang, Shuai Guo, Liyun Feng, Jin Peng, Jian Wang, Jinjie Gu, Junwei Liu

TL;DR

MLB addresses the gap between static medical knowledge benchmarks and real-world clinical utility by introducing a scenario-driven, multi-dimensional evaluation framework for large language models in healthcare. It combines 22 datasets (17 proprietary, 5 public) across 64 subspecialties, expert curation by 300 physicians, and a scalable SFT-trained judge to handle open-ended tasks. Evaluations of 10 leading LLMs reveal a translational gap: models excel on structured tasks like information extraction but struggle with patient-facing reasoning; safety alignment can be strong in smaller models, underscoring the need for targeted training. MLB provides a reproducible path toward clinically viable medical AI, guiding future scenario-based training, evaluation metrics, and deployment practices for safer, more effective healthcare AI.

Abstract

The proliferation of Large Language Models (LLMs) presents transformative potential for healthcare, yet practical deployment is hindered by the absence of frameworks that assess real-world clinical utility. Existing benchmarks test static knowledge, failing to capture the dynamic, application-oriented capabilities required in clinical practice. To bridge this gap, we introduce a Medical LLM Benchmark MLB, a comprehensive benchmark evaluating LLMs on both foundational knowledge and scenario-based reasoning. MLB is structured around five core dimensions: Medical Knowledge (MedKQA), Safety and Ethics (MedSE), Medical Record Understanding (MedRU), Smart Services (SmartServ), and Smart Healthcare (SmartCare). The benchmark integrates 22 datasets (17 newly curated) from diverse Chinese clinical sources, covering 64 clinical specialties. Its design features a rigorous curation pipeline involving 300 licensed physicians. Besides, we provide a scalable evaluation methodology, centered on a specialized judge model trained via Supervised Fine-Tuning (SFT) on expert annotations. Our comprehensive evaluation of 10 leading models reveals a critical translational gap: while the top-ranked model, Kimi-K2-Instruct (77.3% accuracy overall), excels in structured tasks like information extraction (87.8% accuracy in MedRU), performance plummets in patient-facing scenarios (61.3% in SmartServ). Moreover, the exceptional safety score (90.6% in MedSE) of the much smaller Baichuan-M2-32B highlights that targeted training is equally critical. Our specialized judge model, trained via SFT on a 19k expert-annotated medical dataset, achieves 92.1% accuracy, an F1-score of 94.37%, and a Cohen's Kappa of 81.3% for human-AI consistency, validating a reproducible and expert-aligned evaluation protocol. MLB thus provides a rigorous framework to guide the development of clinically viable LLMs.

MLB: A Scenario-Driven Benchmark for Evaluating Large Language Models in Clinical Applications

TL;DR

MLB addresses the gap between static medical knowledge benchmarks and real-world clinical utility by introducing a scenario-driven, multi-dimensional evaluation framework for large language models in healthcare. It combines 22 datasets (17 proprietary, 5 public) across 64 subspecialties, expert curation by 300 physicians, and a scalable SFT-trained judge to handle open-ended tasks. Evaluations of 10 leading LLMs reveal a translational gap: models excel on structured tasks like information extraction but struggle with patient-facing reasoning; safety alignment can be strong in smaller models, underscoring the need for targeted training. MLB provides a reproducible path toward clinically viable medical AI, guiding future scenario-based training, evaluation metrics, and deployment practices for safer, more effective healthcare AI.

Abstract

The proliferation of Large Language Models (LLMs) presents transformative potential for healthcare, yet practical deployment is hindered by the absence of frameworks that assess real-world clinical utility. Existing benchmarks test static knowledge, failing to capture the dynamic, application-oriented capabilities required in clinical practice. To bridge this gap, we introduce a Medical LLM Benchmark MLB, a comprehensive benchmark evaluating LLMs on both foundational knowledge and scenario-based reasoning. MLB is structured around five core dimensions: Medical Knowledge (MedKQA), Safety and Ethics (MedSE), Medical Record Understanding (MedRU), Smart Services (SmartServ), and Smart Healthcare (SmartCare). The benchmark integrates 22 datasets (17 newly curated) from diverse Chinese clinical sources, covering 64 clinical specialties. Its design features a rigorous curation pipeline involving 300 licensed physicians. Besides, we provide a scalable evaluation methodology, centered on a specialized judge model trained via Supervised Fine-Tuning (SFT) on expert annotations. Our comprehensive evaluation of 10 leading models reveals a critical translational gap: while the top-ranked model, Kimi-K2-Instruct (77.3% accuracy overall), excels in structured tasks like information extraction (87.8% accuracy in MedRU), performance plummets in patient-facing scenarios (61.3% in SmartServ). Moreover, the exceptional safety score (90.6% in MedSE) of the much smaller Baichuan-M2-32B highlights that targeted training is equally critical. Our specialized judge model, trained via SFT on a 19k expert-annotated medical dataset, achieves 92.1% accuracy, an F1-score of 94.37%, and a Cohen's Kappa of 81.3% for human-AI consistency, validating a reproducible and expert-aligned evaluation protocol. MLB thus provides a rigorous framework to guide the development of clinically viable LLMs.
Paper Structure (27 sections, 4 figures, 5 tables)

This paper contains 27 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The MLB Benchmark
  4. Evaluation Dimensions
  5. Dataset Composition
  6. Dataset Curation Process
  7. Dataset Preparation
  8. Public Datasets
  9. Proprietary Dataset Construction
  10. Clinical Department Annotation
  11. Expert Review and Refinement
  12. Difficulty Classification
  13. Sample Selection
  14. Evaluation Methodology
  15. Dynamic Evaluation for Bias Mitigation
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Overview of the MLB Benchmark. This figure illustrates the benchmark's hierarchical structure, divided into Fundamental Capabilities (FundCap) and Scenario-based Capabilities (SceneCap), which are further broken down into five core dimensions (MedKQA, MedSE, MedRU, SmartServ, SmartCare).
  • Figure 2: Pipeline of Dataset Curation and Evaluation. The diagram outlines the end-to-end process for creating and evaluating the MLB. It begins with data acquisition from diverse sources (e.g., EHRs, web-based dialogues), proceeds through a multi-paradigm generation pipeline (P1-P4), and undergoes rigorous expert review by 300 licensed physicians. The evaluation phase uses a hybrid scoring protocol in response judgement, culminating in our SFT-based judge model for resolving ambiguities.
  • Figure 3: Overview of the dataset composition, including total sample size, distribution across dimensions, average question length, and difficulty level. The chart displays the distribution of MLB's 22 datasets across the five core dimensions. It highlights the total sample size (4250 samples), the source of data (17 proprietary, 5 public), and the average token length of questions, indicating the complexity of the tasks.
  • Figure 4: Bar chart comparing the performance of selected models on the SmartServ sub-dimension of the MLB.