Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making

Mingyu Derek Ma, Chenchen Ye, Yu Yan, Xiaoxuan Wang, Peipei Ping, Timothy S Chang, Wei Wang

TL;DR

A zero-shot evaluation of leading LLMs is conducted to assess their proficiency in clinical decision-making, shedding light on the potential and limitations of current LLMs in clinical settings, providing valuable insights for future advancements in LLM-powered healthcare.

Abstract

The integration of Artificial Intelligence (AI), especially Large Language Models (LLMs), into the clinical diagnosis process offers significant potential to improve the efficiency and accessibility of medical care. While LLMs have shown some promise in the medical domain, their application in clinical diagnosis remains underexplored, especially in real-world clinical practice, where highly sophisticated, patient-specific decisions need to be made. Current evaluations of LLMs in this field are often narrow in scope, focusing on specific diseases or specialties and employing simplified diagnostic tasks. To bridge this gap, we introduce CliBench, a novel benchmark developed from the MIMIC IV dataset, offering a comprehensive and realistic assessment of LLMs' capabilities in clinical diagnosis. This benchmark not only covers diagnoses from a diverse range of medical cases across various specialties but also incorporates tasks of clinical significance: treatment procedure identification, lab test ordering and medication prescriptions. Supported by structured output ontologies, CliBench enables a precise and multi-granular evaluation, offering an in-depth understanding of LLM's capability on diverse clinical tasks of desired granularity. We conduct a zero-shot evaluation of leading LLMs to assess their proficiency in clinical decision-making. Our preliminary results shed light on the potential and limitations of current LLMs in clinical settings, providing valuable insights for future advancements in LLM-powered healthcare.

CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making

TL;DR

A zero-shot evaluation of leading LLMs is conducted to assess their proficiency in clinical decision-making, shedding light on the potential and limitations of current LLMs in clinical settings, providing valuable insights for future advancements in LLM-powered healthcare.

Abstract

The integration of Artificial Intelligence (AI), especially Large Language Models (LLMs), into the clinical diagnosis process offers significant potential to improve the efficiency and accessibility of medical care. While LLMs have shown some promise in the medical domain, their application in clinical diagnosis remains underexplored, especially in real-world clinical practice, where highly sophisticated, patient-specific decisions need to be made. Current evaluations of LLMs in this field are often narrow in scope, focusing on specific diseases or specialties and employing simplified diagnostic tasks. To bridge this gap, we introduce CliBench, a novel benchmark developed from the MIMIC IV dataset, offering a comprehensive and realistic assessment of LLMs' capabilities in clinical diagnosis. This benchmark not only covers diagnoses from a diverse range of medical cases across various specialties but also incorporates tasks of clinical significance: treatment procedure identification, lab test ordering and medication prescriptions. Supported by structured output ontologies, CliBench enables a precise and multi-granular evaluation, offering an in-depth understanding of LLM's capability on diverse clinical tasks of desired granularity. We conduct a zero-shot evaluation of leading LLMs to assess their proficiency in clinical decision-making. Our preliminary results shed light on the potential and limitations of current LLMs in clinical settings, providing valuable insights for future advancements in LLM-powered healthcare.
Paper Structure (38 sections, 4 figures, 13 tables)

This paper contains 38 sections, 4 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. CliBench: multifaceted benchmark for clinical decisions
  4. Realistic clinical decision tasks with expert-defined output spaces
  5. Data processing and clinical data elements extraction
  6. Balanced and diverse evaluation instances sampling
  7. Prompt construction
  8. Matching from free text generation to medical code candidate
  9. Multi-granular evaluation metrics
  10. Statistics of the evaluation set
  11. Comparing methods and models
  12. Performance of LLMs on clinical decisions
  13. Discharge diagnoses
  14. Procedure, lab orders and prescriptions
  15. Analysis
  16. ...and 23 more sections

Figures (4)

  • Figure 1: The data distribution of the evaluation set for the diagnosis decision task.
  • Figure 2: The average F1 score of GPT-4o and LLaMA3 70B Instruct by patient attributes.
  • Figure 3: The breakdown diagnosis performance for full code level by the difficulties of the patient cases, specifically: (a) how long the patient stayed in the hospital, (b) the number of unique diagnosis chapters in ground-truth billing diagnosis codes, (c) the number of diagnosis codes at discharge.
  • Figure 4: Distribution of length of input prompt for clinical decision tasks using LLaMA3 default tokenizer.