Small Language Models for Emergency Departments Decision Support: A Benchmark Study

TL;DR

The paper tackles the challenge of enabling effective decision support in emergency departments under hardware and privacy constraints by benchmarking small language models with $\le 8\text{B}$ parameters on four ED-relevant tasks: MedMCQA, MedQA-4Options, PubMedQA, and Medical Abstracts. It systematically compares general-domain versus medical-domain SLMs using quantization-aware evaluation and practical deployment considerations, finding that general-domain models often outperform medical-tuned counterparts in ED contexts. The study demonstrates that strong instruction-following and broad pretraining can yield robust performance for ED tasks, with dedicated summarization requiring chat-tuned models. The results inform practical guidance for on-site, private ED AI deployments, including quantization strategies, hardware choices, and a modular, multi-agent deployment approach, while highlighting the need for future domain-specific fine-tuning and real-world pilots.

Abstract

Large language models (LLMs) have become increasingly popular in medical domains to assist physicians with a variety of clinical and operational tasks. Given the fast-paced and high-stakes environment of emergency departments (EDs), small language models (SLMs), characterized by a reduction in parameter count compared to LLMs, offer significant potential due to their inherent reasoning capability and efficient performance. This enables SLMs to support physicians by providing timely and accurate information synthesis, thereby improving clinical decision-making and workflow efficiency. In this paper, we present a comprehensive benchmark designed to identify SLMs suited for ED decision support, taking into account both specialized medical expertise and broad general problem-solving capabilities. In our evaluations, we focus on SLMs that have been trained on a mixture of general-domain and medical corpora. A key motivation for emphasizing SLMs is the practical hardware limitations, operational cost constraints, and privacy concerns in the typical real-world deployments. Our benchmark datasets include MedMCQA, MedQA-4Options, and PubMedQA, with the medical abstracts dataset emulating tasks aligned with real ED physicians' daily tasks. Experimental results reveal that general-domain SLMs surprisingly outperform their medically fine-tuned counterparts across these diverse benchmarks for ED. This indicates that for ED, specialized medical fine-tuning of the model may not be required.