What Does Neuro Mean to Cardio? Investigating the Role of Clinical Specialty Data in Medical LLMs

TL;DR

This work introduces S-MedQA, the first English medical QA dataset with fine-grained clinical specialty annotations across 15 specialties, created by augmenting MedQA and MedMCQA with machine-assisted labeling and expert validation. Through single- and multi-label annotation via prompts, voting, and conformal prediction, the authors enable cross-disciplinary QA evaluation and analyze how fine-tuning on specialty data affects performance. Key findings show that cross-specialty transfer often outperforms in-domain fine-tuning, and that improvements stem mainly from domain shifting (general to medical) rather than injection of specialty-specific knowledge, as evidenced by token-probability shifts and overlap analyses. The work provides a valuable resource for benchmarking medical LLMs, informs data-selection strategies for fine-tuning, and suggests that future work should explore domain-focused pretraining and cross-domain transfer to better capture clinical knowledge.

Abstract

In this paper, we introduce S-MedQA, an English medical question-answering (QA) dataset for benchmarking large language models (LLMs) in fine-grained clinical specialties. S-MedQA has over 20k examples, covers 15 medical specialties, and QA pairs can have multiple specialty annotations (e.g., when a question is cross-disciplinary), constructed with both machine and expert verification to maximize data availability. We use S-MedQA to investigate the role of clinical specialty data in the knowledge-intensive scenario of medical QA. Our results show that 1) training on data from a clinical specialty does not necessarily lead to best performance on that specialty, and 2) regardless of the specialty the LLM was fine-tuned on, token probabilities of clinically relevant terms increase consistently across all specialties. Thus, we hypothesize improvement gains are derived mostly from domain shifting (e.g., general to medical) rather than specialty-specific knowledge injection, and suggest rethinking the role of fine-tuning data in the medical domain.

Figures (10)

• Figure 1: Overview of S-MedQA's construction process. For single specialty annotation of each sample, we generate predictions using 5 different prompts and only keep those where predictions agree ($3+$, $4+$, or $5$ times). A medical expert manually annotates S-MedQA's entire validation and test set. We randomly sample $1,000$ questions from our train set and ask multiple medical experts to evaluate GPT-3.5's predictions (we detail inter-annotator agreement in § \ref{['sec:manual_validation']}), achieving accuracies ranging from $90.8$--$97.8$%. For multi-specialty annotation, we leverage conformal prediction to assign labels across multiple clinical specialties, using a held-out set of $300$ samples manually annotated by medical experts as calibration/test set. It achieves a precision of $0.69$, recall of $0.52$, with $24\%$ exactly correct matches and an average prediction length of $1.55$.
• Figure 2: The distribution of all specialties classified by GPT-3.5. The dark blue specialties are the 15 we finally included in our benchmark. Note that we conduct experiments on the 6 common specialties for simplicity.
• Figure 3: The illustration of first token probability, string matching, and our approach (classifier) to evaluating LLMs performance on S-MedQA. We use text output instead of first token probability for evaluation because the latter suffers heavily from selection bias in multiple-choice QA wang2024my. However, string matching does not work in some cases. Our classifier trained on Mistral-v0.2 works successfully with an accuracy of 96.5%.
• Figure 4: Heatmap showing the overlap of clinical terms within and across clinical specialties between train and test QA pairs in S-MedQA.
• Figure 5: Negative log-probabilities for clinically relevant tokens between baseline Llama-3.1-8B-Instruct and the same model further fine-tuned on each specialty data. Each group represents tokens categorized into different clinical specialties. Each color means that the same model is further fine-tuned on each specialty data.