Clinical Prompt Learning with Frozen Language Models

TL;DR

This work investigates applying prompt learning to clinical NLP tasks using a frozen domain-specific PLM (Bio-ClinicalBERT) and compares it against traditional fine-tuning across four MIMIC-III-derived tasks. Prompt learning, particularly with mixed templates and soft verbalizers, achieves competitive or superior performance to fine-tuning while requiring far fewer trainable parameters, especially in few-shot and frozen-PLM scenarios. The findings suggest a practical path toward resource-efficient, interpretable clinical decision-support tools that can operate on modest hardware. Overall, the study demonstrates that prompt-based approaches can approach or surpass state-of-the-art performance with substantially reduced computational demands in clinical settings.

Abstract

Prompt learning is a new paradigm in the Natural Language Processing (NLP) field which has shown impressive performance on a number of natural language tasks with common benchmarking text datasets in full, few-shot, and zero-shot train-evaluation setups. Recently, it has even been observed that large but frozen pre-trained language models (PLMs) with prompt learning outperform smaller but fine-tuned models. However, as with many recent NLP trends, the performance of even the largest PLMs such as GPT-3 do not perform well on specialized domains (e.g. medical text), and the common practice to achieve State of the Art (SoTA) results still consists of pre-training and fine-tuning the PLMs on downstream tasks. The reliance on fine-tuning large PLMs is problematic in clinical settings where data is often held in non-GPU environments, and more resource efficient methods of training specialized domain models is crucial. We investigated the viability of prompt learning on clinically meaningful decision tasks and directly compared with more traditional fine-tuning methods. Results are partially in line with the prompt learning literature, with prompt learning able to match or improve on traditional fine-tuning with substantially fewer trainable parameters and requiring less training data. We argue that prompt learning therefore provides lower computational resource costs applicable to clinical settings, that can serve as an alternative to fine-tuning ever increasing in size PLMs. Complementary code to reproduce experiments presented in this work can be found at: https://github.com/NtaylorOX/Public_Clinical_Prompt.

Figures (6)

• Figure 1: Illustration of conventional fine-tuning method, with an option to freeze the PLM, shown in dotted line. Here [CLS] and [SEP] tokens are special tokens for BERT-based models that are added to the beginning and end of sequences.
• Figure 2: Illustration of manual template and verbalizer in prompt learning.
• Figure 3: Illustration of soft template and verbalizer in prompt learning. If the <[soft]> token $P_2$ is not defined manually in advance, the embedding $h(P_2) \in \mathbb{R}^m$ will be randomly initialized in the hidden space.
• Figure 4: Balanced accuracy for prompt learning and traditional fine-tuning frameworks across the four clinical tasks. "LoS" refers to length of stay and "Full" refers to a full data set size which varies from task to task.
• Figure 5: Balanced accuracy for prompt learning versus traditional fine-tuning across increasing number of trainable parameters with frozen PLM. For readability, logarithmic scale is used for $x$-axis.