Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimizing Medical Question-Answering Systems: A Comparative Study of Fine-Tuned and Zero-Shot Large Language Models with RAG Framework

Tasnimul Hassan, Md Faisal Karim, Haziq Jeelani, Elham Behnam, Robert Green, Fayeq Jeelani Syed

TL;DR

The paper tackles factual accuracy and hallucinations in medical QA by integrating retrieval-augmented generation with LoRA-fine-tuned open-source LLMs. It grounds answers in retrieved biomedical literature via a Dense Passage Retriever, improving reliability on PubMedQA and MedMCQA. The LLaMA 2 and Falcon models achieve substantial gains over zero-shot baselines, with LLaMA 2 + RAG reaching 71.8% PubMedQA accuracy and reduced unsupported content. The work discusses efficiency, transparency, and deployment considerations, and points to future clinician-in-the-loop and multi-modal extensions.

Abstract

Medical question-answering (QA) systems can benefit from advances in large language models (LLMs), but directly applying LLMs to the clinical domain poses challenges such as maintaining factual accuracy and avoiding hallucinations. In this paper, we present a retrieval-augmented generation (RAG) based medical QA system that combines domain-specific knowledge retrieval with open-source LLMs to answer medical questions. We fine-tune two state-of-the-art open LLMs (LLaMA~2 and Falcon) using Low-Rank Adaptation (LoRA) for efficient domain specialization. The system retrieves relevant medical literature to ground the LLM's answers, thereby improving factual correctness and reducing hallucinations. We evaluate the approach on benchmark datasets (PubMedQA and MedMCQA) and show that retrieval augmentation yields measurable improvements in answer accuracy compared to using LLMs alone. Our fine-tuned LLaMA~2 model achieves 71.8% accuracy on PubMedQA, substantially improving over the 55.4% zero-shot baseline, while maintaining transparency by providing source references. We also detail the system design and fine-tuning methodology, demonstrating that grounding answers in retrieved evidence reduces unsupported content by approximately 60%. These results highlight the potential of RAG-augmented open-source LLMs for reliable biomedical QA, pointing toward practical clinical informatics applications.

Optimizing Medical Question-Answering Systems: A Comparative Study of Fine-Tuned and Zero-Shot Large Language Models with RAG Framework

TL;DR

The paper tackles factual accuracy and hallucinations in medical QA by integrating retrieval-augmented generation with LoRA-fine-tuned open-source LLMs. It grounds answers in retrieved biomedical literature via a Dense Passage Retriever, improving reliability on PubMedQA and MedMCQA. The LLaMA 2 and Falcon models achieve substantial gains over zero-shot baselines, with LLaMA 2 + RAG reaching 71.8% PubMedQA accuracy and reduced unsupported content. The work discusses efficiency, transparency, and deployment considerations, and points to future clinician-in-the-loop and multi-modal extensions.

Abstract

Medical question-answering (QA) systems can benefit from advances in large language models (LLMs), but directly applying LLMs to the clinical domain poses challenges such as maintaining factual accuracy and avoiding hallucinations. In this paper, we present a retrieval-augmented generation (RAG) based medical QA system that combines domain-specific knowledge retrieval with open-source LLMs to answer medical questions. We fine-tune two state-of-the-art open LLMs (LLaMA~2 and Falcon) using Low-Rank Adaptation (LoRA) for efficient domain specialization. The system retrieves relevant medical literature to ground the LLM's answers, thereby improving factual correctness and reducing hallucinations. We evaluate the approach on benchmark datasets (PubMedQA and MedMCQA) and show that retrieval augmentation yields measurable improvements in answer accuracy compared to using LLMs alone. Our fine-tuned LLaMA~2 model achieves 71.8% accuracy on PubMedQA, substantially improving over the 55.4% zero-shot baseline, while maintaining transparency by providing source references. We also detail the system design and fine-tuning methodology, demonstrating that grounding answers in retrieved evidence reduces unsupported content by approximately 60%. These results highlight the potential of RAG-augmented open-source LLMs for reliable biomedical QA, pointing toward practical clinical informatics applications.

Paper Structure

This paper contains 12 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. System Overview
  5. Fine-Tuning the LLMs with LoRA
  6. Results and Discussion
  7. Quantitative Performance
  8. Efficiency and Resource Considerations
  9. Comparison with Prior Works
  10. Hallucination Reduction and Interpretability
  11. Clinical and Deployment Considerations
  12. Conclusion

Figures (1)

  • Figure 1: High-level architecture of the proposed RAG-based medical QA system. The system first retrieves relevant documents from a biomedical knowledge repository given a user question. A fine-tuned LLM (either LLaMA 2 or Falcon) then generates an answer conditioned on the question and retrieved context. Grounding the answer in retrieved evidence helps reduce hallucinations and improve accuracy.