ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?

TL;DR

ClinicalBench systematically benchmarks a wide range of general-purpose and medical LLMs against traditional ML models on three real-world clinical prediction tasks using MIMIC-III and MIMIC-IV. Across prompting, prompting-engineering, and fine-tuning, the study finds that LLMs do not yet outperform conventional models like XGBoost or SVM in Length-of-Stay, Mortality, or Readmission prediction, highlighting limitations in clinical reasoning and decision-making. The work reveals limited gains from decoding temperature, model scaling, or prompting strategies, and only sporadic improvements from fine-tuning; still, traditional models maintain a robust advantage. The authors argue for cautious deployment of LLMs in clinical settings and propose ClinicalBench as a benchmark to guide future development toward real-world clinical practice and safety.

Abstract

Large Language Models (LLMs) hold great promise to revolutionize current clinical systems for their superior capacities on medical text processing tasks and medical licensing exams. Meanwhile, traditional ML models such as SVM and XGBoost have still been mainly adopted in clinical prediction tasks. An emerging question is Can LLMs beat traditional ML models in clinical prediction? Thus, we build a new benchmark ClinicalBench to comprehensively study the clinical predictive modeling capacities of both general-purpose and medical LLMs, and compare them with traditional ML models. ClinicalBench embraces three common clinical prediction tasks, two databases, 14 general-purpose LLMs, 8 medical LLMs, and 11 traditional ML models. Through extensive empirical investigation, we discover that both general-purpose and medical LLMs, even with different model scales, diverse prompting or fine-tuning strategies, still cannot beat traditional ML models in clinical prediction yet, shedding light on their potential deficiency in clinical reasoning and decision-making. We call for caution when practitioners adopt LLMs in clinical applications. ClinicalBench can be utilized to bridge the gap between LLMs' development for healthcare and real-world clinical practice.

Figures (29)

• Figure 1: Overview of ClinicalBench.
• Figure 2: Examples of Confusion Matrix of LLMs and Traditional ML Models on Mortality Prediction. Experiments are conducted in MIMIC-III dataset. The complete confusion matrices across different methods, tasks and datasets are in Appendix \ref{['Confusion Matrix of Traditional ML Models and LLMs']}.
• Figure 3: Performance Comparison Between LLMs with Different Temperatures and Traditional ML Models on Length-of-Stay Prediction. Experiments are conducted in MIMIC-III dataset. More results on Mortality and Readmission Prediction are in Appenidix \ref{['Results of LLMs\nwith Different Temperatures of Decoding']}.
• Figure 4: Performance Comparison Between Fine-tuned LLMs and Traditional ML Models on Clinical Prediction Tasks. Experiments are conducted on sampled MIMIC-III and MIMIC-IV datasets. Length-of-Stay Prediction adopts Macro F1$\%$ and the others use F1$\%$ as the metric. LoRA (Full) and LoRA (Last Layer) refer to applying LoRA to full layers and only last layer respectively.
• Figure 5: Performance Comparison Between LLMs with Different Temperatures and Traditional ML Models on Mortality Prediction and Readmission Prediction. Experiments are conducted in MIMIC-III dataset.