TOP-Training: Target-Oriented Pretraining for Medical Extractive Question Answering

TL;DR

The paper addresses Medical-EQA under data scarcity and autoregressive LLM hallucinations by introducing TOP-Training, a target-oriented pretraining framework that builds synthetic, target-specific contexts from extracted entities and uses bidirectional encoders. The approach pretrains on this synthetic corpus and then performs two rounds of fine-tuning (SQuAD, then the target Medical-EQA dataset), enabling strong, data-efficient adaptation. Empirical results on COVID-QA and RadQA show TOP-Training achieving state-of-the-art or competitive performance with modest synthetic data, and robustness across encoder architectures and data-generation settings. The work highlights limitations of autoregressive LLMs for extractive QA and suggests broader domain-adaptation implications, while acknowledging computational cost and hallucination as avenues for future work.

Abstract

We study extractive question-answering in the medical domain (Medical-EQA). This problem has two main challenges: (i) domain specificity, as most AI models lack necessary domain knowledge, and (ii) extraction-based answering style, which restricts most autoregressive LLMs due to potential hallucinations. To handle those challenges, we propose TOP-Training, a target-oriented pre-training paradigm that stands out among all domain adaptation techniques with two desirable features: (i) TOP-Training moves one step further than popular domain-oriented fine-tuning since it not only moves closer to the target domain, but also familiarizes itself with the target dataset, and (ii) it does not assume the existence of a large set of unlabeled instances from the target domain. Specifically, for a target Medical-EQA dataset, we extract its entities and leverage large language models (LLMs) to generate synthetic texts containing those entities; we then demonstrate that pretraining on this synthetic text data yields better performance on the target Medical-EQA benchmarks. Overall, our contributions are threefold: (i) TOP-Training, a new pretraining technique to effectively adapt LLMs to better solve a target problem, (ii) TOP-Training has a wide application scope because it does not require the target problem to have a large set of unlabeled data, and (iii) our experiments highlight the limitations of autoregressive LLMs, emphasizing TOP-Training as a means to unlock the true potential of bidirectional LLMs.

Figures (7)

• Figure 1: TOP-Training. First, we extract relevant entities from the target dataset to generate our synthetic pre-training corpora. Next, we train an open-domain model on this corpus followed by two rounds of EQA fine-tuning, i.e., first on an open-domain dataset to learn what EQA is as a task and then on the target Medical-EQA dataset.
• Figure 2: Information Leakage Validation Trials (Left - EM | Right - F1): RoBERTa (ours) was trained on a subset of the 47k corpus with entities only from the 80% train set. All of the models were fine-tuned in the usual manner i.e. SQuAD$\rightarrow$COVID-QA (80% train set) and evaluated on the 20% test set.
• Figure 3: COVID-QA positive examples (underline = entity | red = prompt) | olive = generation
• Figure 4: COVID-QA negative examples (underline = entity | red = prompt) | olive = generation
• Figure 5: RadQA - normal prompt (underline = entity | red = prompt) | olive = generation