TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verification

TL;DR

TAGS tackles the brittleness and adaptability gaps of current medical QA systems by introducing a test-time generalist–specialist framework that combines retrieval-augmented reasoning with a dual-agent collaboration and a verification-based aggregation. It comprises Hierarchical Retrieval Augmentation to supply diverse, rationale-aligned exemplars, Generalist–Specialist Reasoning Collaboration to generate complementary reasoning paths, and Uncertainty-Aware Answer Aggregation to select robust answers based on reasoning consistency. Across nine hard MedQA benchmarks and multiple base models, TAGS consistently outperforms prompting-based, retrieval-only, and some fine-tuned baselines, even improving a 7B model to competitive levels without parameter updates. The approach demonstrates that structured, inference-time collaboration, guided retrieval, and verification can close much of the gap between zero-shot and fine-tuned medical LLMs while maintaining adaptability and scalability in real-world clinical QA tasks.

Abstract

Recent advances such as Chain-of-Thought prompting have significantly improved large language models (LLMs) in zero-shot medical reasoning. However, prompting-based methods often remain shallow and unstable, while fine-tuned medical LLMs suffer from poor generalization under distribution shifts and limited adaptability to unseen clinical scenarios. To address these limitations, we present TAGS, a test-time framework that combines a broadly capable generalist with a domain-specific specialist to offer complementary perspectives without any model fine-tuning or parameter updates. To support this generalist-specialist reasoning process, we introduce two auxiliary modules: a hierarchical retrieval mechanism that provides multi-scale exemplars by selecting examples based on both semantic and rationale-level similarity, and a reliability scorer that evaluates reasoning consistency to guide final answer aggregation. TAGS achieves strong performance across nine MedQA benchmarks, boosting GPT-4o accuracy by 13.8%, DeepSeek-R1 by 16.8%, and improving a vanilla 7B model from 14.1% to 23.9%. These results surpass several fine-tuned medical LLMs, without any parameter updates. The code will be available at https://github.com/JianghaoWu/TAGS.

Figures (2)

• Figure 1: Overview of the proposed TAGS framework. The architecture consists of three modules: (A) HRA (Hierarchical Retrieval Augmentation), a two-stage retrieval process that selects semantically relevant exemplars ($\mathcal{T}_1$) and refines them based on rationale alignment ($\mathcal{T}_{G,2}$, $\mathcal{T}_{S,2}$). (B) GSRC (Generalist-Specialist Reasoning Collaboration) employs dual-agent reasoning across two rounds, generating four candidate (Rationale, Answer) pairs. (C) UAAA (Uncertainty-Aware Answer Aggregation) assesses rationale consistency using the RCE and aggregates reliability scores ($c$) to determine the final answer.
• Figure 2: Hyper-parameter sensitivity analysis of specialist count and retrieval size in relation to accuracy.