Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering
Feras AlMannaa, Talia Tseriotou, Jenny Chim, Maria Liakata
TL;DR
The paper tackles long-context, patient-centric medical QA by evaluating LLMs of varying sizes with and without retrieval-augmented generation (RAG) across MC, extractive, and open-ended tasks on longitudinal EHR data from MIMIC datasets. It introduces a hybrid RAG pipeline with dense and sparse retrieval, ranking, and temporal document ordering, paired with a multi-metric evaluation framework that includes LLM-as-a-judge. Key findings show larger models and carefully tuned RAG setups yield gains over full-document prompting in several settings, though performance remains sensitive to noise, task formulation, and evaluation metrics. The work informs practical deployment considerations for longitudinal QA and highlights directions for scalable multi-note reasoning and robust clinical evaluation.
Abstract
This study is the first to investigate LLM comprehension capabilities over long-context (LC) medical QA of clinical relevance. Our comprehensive assessment spans a range of content-inclusion settings based on their relevance, LLM models of varying capabilities and datasets across task formulations, revealing insights on model size effects, limitations, underlying memorization issues and the benefits of reasoning models. Importantly, we examine the effect of RAG on medical LC comprehension, uncover best settings in single versus multi-document reasoning datasets and showcase RAG strategies for improvements over LC. We shed light into some of the evaluation aspects using a multi-faceted approach. Our qualitative and error analyses address open questions on when RAG is beneficial over LC, revealing common failure cases.