MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarization

TL;DR

MEDVOC introduces a dynamic vocabulary adaptation framework for fine-tuning encoder-decoder PLMs on medical text summarization, treating vocabulary as an optimizable parameter and dramatically speeding up the vocabulary-adaptation process via a fragment-score guided search. The method constructs a MEDVOC vocabulary through candidate subword generation, target-domain overlap, and controlled expansion, coupled with an intermediate fine-tuning step on biomedical titles (PAC) to align models to the medical domain. Across four medical datasets and three PLMs, MEDVOC achieves substantial Rouge-L and Concept Score gains, excelling in zero-shot, long-form, and high-OOV conditions, and is validated by human experts as more faithful. The approach generalizes across PLMs and significantly reduces hyperparameter search time, offering a practical path to deploy vocabulary adaptation for medical summarization in real-world settings.

Abstract

This work presents a dynamic vocabulary adaptation strategy, MEDVOC, for fine-tuning pre-trained language models (PLMs) like BertSumAbs, BART, and PEGASUS for improved medical text summarization. In contrast to existing domain adaptation approaches in summarization, MEDVOC treats vocabulary as an optimizable parameter and optimizes the PLM vocabulary based on fragment score conditioned only on the downstream task's reference summaries. Unlike previous works on vocabulary adaptation (limited only to classification tasks), optimizing vocabulary based on summarization tasks requires an extremely costly intermediate fine-tuning step on large summarization datasets. To that end, our novel fragment score-based hyperparameter search very significantly reduces this fine-tuning time -- from 450 days to less than 2 days on average. Furthermore, while previous works on vocabulary adaptation are often primarily tied to single PLMs, MEDVOC is designed to be deployable across multiple PLMs (with varying model vocabulary sizes, pre-training objectives, and model sizes) -- bridging the limited vocabulary overlap between the biomedical literature domain and PLMs. MEDVOC outperforms baselines by 15.74% in terms of Rouge-L in zero-shot setting and shows gains of 17.29% in high Out-Of-Vocabulary (OOV) concentrations. Our human evaluation shows MEDVOC generates more faithful medical summaries (88% compared to 59% in baselines). We make the codebase publicly available at https://github.com/gb-kgp/MEDVOC.

Figures (5)

• Figure 1: Illustrative example of BertSumAbs model from EBM dataset. Purple text color: indicates semantic or surface form overlaps with RS, Bold text: indicates medical (UMLS) concept-bearing words, Yellow highlight: OOV words that are ultimately added to the updated vocabulary, and Orange highlight: medical concept-bearing word(s) that overlap with reference summary.
• Figure 2: Methodological overview of MEDVOC and existing fine-tuning strategy.
• Figure 3: (a) Heatmap to show vocabulary overlap among different training datasets, computed based on the overlap between the top 10K most frequent words in each dataset. CNN corresponds to CNN/DailyMail dataset. (b) Words in the BioASQ dataset across three PLMs are split into four or more subwords; we observe that most of them are medical terms.
• Figure 4: (a) Zero-shot and few-shot performance in terms of Rouge-L scores averaged across the three PLMs. MEDVOC shows statistically significant improvement over IFT-CNN in most settings, except BioASQ. (b) Performance improvement of MEDVOC over IFT-CNN in high OOV concentration and long-form reference summaries (top-ten percentile); PEGASUS shows the highest jump.
• Figure 5: (a) Shift observed in the positional embedding for BertSumAbs in terms of Euclidean distance. (b) Human evaluation scores comparison over $100$ randomly selected test data points. MEDVOC produces more relevant, coherent, and faithful summaries during human evaluation with medical experts.