Do Clinical Question Answering Systems Really Need Specialised Medical Fine Tuning?

TL;DR

This work challenges the necessity of domain-specific fine-tuning for clinical question answering by proposing MedAssess-X, an inference-time activation-level alignment layer. By injecting lightweight steering vectors at decoding, MedAssess-X stabilizes reasoning across both general-purpose and specialised medical LLMs, mitigating the Specialisation Fallacy. Empirical results show consistent improvements in accuracy and factual consistency while reducing safety errors, with modest computational overhead. The framework relies on task-specific steering and a contrastive vector construction to guide medically grounded reasoning without updating weights. Overall, MedAssess-X offers a practical path toward robust, safe CQA deployments across diverse model families.

Abstract

Clinical Question-Answering (CQA) industry systems are increasingly rely on Large Language Models (LLMs), yet their deployment is often guided by the assumption that domain-specific fine-tuning is essential. Although specialised medical LLMs such as BioBERT, BioGPT, and PubMedBERT remain popular, they face practical limitations including narrow coverage, high retraining costs, and limited adaptability. Efforts based on Supervised Fine-Tuning (SFT) have attempted to address these assumptions but continue to reinforce what we term the SPECIALISATION FALLACY-the belief that specialised medical LLMs are inherently superior for CQA. To address this assumption, we introduce MEDASSESS-X, a deployment-industry-oriented CQA framework that applies alignment at inference time rather than through SFT. MEDASSESS-X uses lightweight steering vectors to guide model activations toward medically consistent reasoning without updating model weights or requiring domain-specific retraining. This inference-time alignment layer stabilises CQA performance across both general-purpose and specialised medical LLMs, thereby resolving the SPECIALISATION FALLACY. Empirically, MEDASSESS-X delivers consistent gains across all LLM families, improving Accuracy by up to +6%, Factual Consistency by +7%, and reducing Safety Error Rate by as much as 50%.

Figures (4)

• Figure 1: Representative failure cases observed in industry CQA systems, including triage assistance, literature summarisation, and patient-facing guidance. These failure highlight how domain-specialised or fine-tuned models often struggle when applied beyond their narrow training scope, leading to (i) rigidity—inability to Specialisation Fallacy--relying solely on medically fine-tuned models does not guarantee reliable CQA performance in real-world deployments. This motivates the need for an inference-time alignment layer such as MedAssess-X, which stabilises reasoning across heterogeneous LLMs without domain-specific retraining.
• Figure 2: MedAssess-X framework operates as an activation-level alignment layer that sits between the base LLM and its final decoding stages. Instead of updating model parameters through SFT, the framework introduces lightweight steering vectors that modulate hidden representations during inference to produce medically consistent reasoning trajectories.
• Figure 3: Decoder-only TRUE/FALSE prompting setup used for decoder-only LLMs. Prediction corresponds to the first generated token ("True" or "False").
• Figure 4: Encoder and encoder--decoder prompting/classification setup used for encoder-decoder only LLMs. T5 models generate a constrained binary token, whereas encoder-only medical models perform TRUE/FALSE classification using their final hidden-state encoder representations.