MedTutor: A Retrieval-Augmented LLM System for Case-Based Medical Education

TL;DR

MedTutor tackles the challenge of transforming clinical case reports into reliable, educational content by grounding LLM generation in a hybrid retrieval pipeline that combines a local textbook database with live literature via PubMed and Semantic Scholar. It employs a dense embedding-based local retrieval (Qwen3-Embedding-8B) and a strong reranker (Qwen3-Reranker-8B) in a retrieval-augmented framework, with generation powered by multi-GPU, batch-oriented vLLM inference and fully local deployment to preserve privacy. The authors conduct a comprehensive evaluation, including expert radiologists assessing upstream and downstream outputs and a large-scale LLM-as-a-Judge study across six LLMs, using 2000 reports per dataset across five radiology corpora, and announce public release of the MedTutor dataset and UI. Key findings show MedGemma-27B generally outperforms generic LLMs on educational content quality and MCQ plausibility, while the LLM-as-a-Judge approach provides scalable but not perfectly calibrated judgments relative to human experts, underscoring the continued need for expert oversight. Overall, MedTutor demonstrates a scalable, evidence-grounded approach to augmenting medical education with case-based content while maintaining privacy and offering a valuable benchmark for future research in trustworthy medical AI systems.

Abstract

The learning process for medical residents presents significant challenges, demanding both the ability to interpret complex case reports and the rapid acquisition of accurate medical knowledge from reliable sources. Residents typically study case reports and engage in discussions with peers and mentors, but finding relevant educational materials and evidence to support their learning from these cases is often time-consuming and challenging. To address this, we introduce MedTutor, a novel system designed to augment resident training by automatically generating evidence-based educational content and multiple-choice questions from clinical case reports. MedTutor leverages a Retrieval-Augmented Generation (RAG) pipeline that takes clinical case reports as input and produces targeted educational materials. The system's architecture features a hybrid retrieval mechanism that synergistically queries a local knowledge base of medical textbooks and academic literature (using PubMed, Semantic Scholar APIs) for the latest related research, ensuring the generated content is both foundationally sound and current. The retrieved evidence is filtered and ordered using a state-of-the-art reranking model and then an LLM generates the final long-form output describing the main educational content regarding the case-report. We conduct a rigorous evaluation of the system. First, three radiologists assessed the quality of outputs, finding them to be of high clinical and educational value. Second, we perform a large scale evaluation using an LLM-as-a Judge to understand if LLMs can be used to evaluate the output of the system. Our analysis using correlation between LLMs outputs and human expert judgments reveals a moderate alignment and highlights the continued necessity of expert oversight.

Figures (9)

• Figure 1: The overall architecture of the MedTutor system.
• Figure 2: Pairwise Cohen's Kappa ($\kappa$) scores for Upstream Tasks. This figure shows the agreement between three pairs of annotators for keyword appropriateness and paper relevance.
• Figure 3: Pairwise Cohen's Kappa ($\kappa$) scores for Llama3.3-70B-Instruct Generated Content.
• Figure 4: Pairwise Cohen's Kappa ($\kappa$) scores for MedGemma-27B-text-it Generated Content.
• Figure 5: The MedTutor UI (Part 1 of 2): Main dashboard and initial configuration settings.