Supervised Fine-Tuning or In-Context Learning? Evaluating LLMs for Clinical NER

TL;DR

The paper investigates clinical NER on the CADEC corpus by comparing domain-specific and general BERT-style encoders with GPT-4o-based approaches in both in-context learning and supervised fine-tuning. It finds that SFT with GPT-4o delivers the strongest overall performance (about 0.871 F1) at a high cost, while simple in-context prompts outperform more elaborate prompts and can even outperform some fine-tuning configurations. Domain-specific encoders offer limited gains over strong general baselines, and LLMs tend to perform best on simplified, two-label tasks. These results inform practical trade-offs between accuracy and compute when deploying clinical NER in real-world settings, and point to future directions involving hyperparameter tuning, open-source LLMs, and model ensembles.

Abstract

We study clinical Named Entity Recognition (NER) on the CADEC corpus and compare three families of approaches: (i) BERT-style encoders (BERT Base, BioClinicalBERT, RoBERTa-large), (ii) GPT-4o used with few-shot in-context learning (ICL) under simple vs.\ complex prompts, and (iii) GPT-4o with supervised fine-tuning (SFT). All models are evaluated on standard NER metrics over CADEC's five entity types (ADR, Drug, Disease, Symptom, Finding). RoBERTa-large and BioClinicalBERT offer limited improvements over BERT Base, showing the limit of these family of models. Among LLM settings, simple ICL outperforms a longer, instruction-heavy prompt, and SFT achieves the strongest overall performance (F1 $\approx$ 87.1%), albeit with higher cost. We find that the LLM achieve higher accuracy on simplified tasks, restricting classification to two labels.

Figures (3)

• Figure 1: Distribution of the length (expressed as the number of words) of the posts that were included in CADEC.
• Figure 2: Distribution of the IOB entity labels in CADEC.
• Figure 3: Top 20 most common terms in CADEC.