Domain-Specific Knowledge Graphs in RAG-Enhanced Healthcare LLMs

TL;DR

This work investigates how domain-specific knowledge graphs derived from PubMed abstracts can ground retrieval-augmented generation for biomedical QA. It constructs three graphs—$\mathbb{G}_1$ (T2DM-focused), $\mathbb{G}_2$ (AD-focused), and $\mathbb{G}_3$ (AD+T2DM)—and evaluates two probes to test scope alignment and multi-hop reasoning across seven instruction-tuned LLMs at three temperatures. The key finding is that precision-aligned retrieval, particularly using $\mathbb{G}_2$, yields the most consistent gains, while indiscriminate graph unions often introduce distractors and reduce accuracy; larger models sometimes rely on parametric priors and benefit less from KG-RAG on merged tasks, whereas smaller models gain more from well-scoped retrieval. The paper provides practical guidelines for graph selection, model sizing, and retrieval/reranking, and it releases code and data to enable replication. Overall, the work clarifies when KG-RAG adds value in healthcare QA and emphasizes precision over breadth for domain-grounded LLM reasoning.

Abstract

Large Language Models (LLMs) generate fluent answers but can struggle with trustworthy, domain-specific reasoning. We evaluate whether domain knowledge graphs (KGs) improve Retrieval-Augmented Generation (RAG) for healthcare by constructing three PubMed-derived graphs: $\mathbb{G}_1$ (T2DM), $\mathbb{G}_2$ (Alzheimer's disease), and $\mathbb{G}_3$ (AD+T2DM). We design two probes: Probe 1 targets merged AD T2DM knowledge, while Probe 2 targets the intersection of $\mathbb{G}_1$ and $\mathbb{G}_2$. Seven instruction-tuned LLMs are tested across retrieval sources {No-RAG, $\mathbb{G}_1$, $\mathbb{G}_2$, $\mathbb{G}_1$ + $\mathbb{G}_2$, $\mathbb{G}_3$, $\mathbb{G}_1$+$\mathbb{G}_2$ + $\mathbb{G}_3$} and three decoding temperatures. Results show that scope alignment between probe and KG is decisive: precise, scope-matched retrieval (notably $\mathbb{G}_2$) yields the most consistent gains, whereas indiscriminate graph unions often introduce distractors that reduce accuracy. Larger models frequently match or exceed KG-RAG with a No-RAG baseline on Probe 1, indicating strong parametric priors, whereas smaller/mid-sized models benefit more from well-scoped retrieval. Temperature plays a secondary role; higher values rarely help. We conclude that precision-first, scope-matched KG-RAG is preferable to breadth-first unions, and we outline practical guidelines for graph selection, model sizing, and retrieval/reranking. Code and Data available here - https://github.com/sydneyanuyah/RAGComparison

Figures (2)

• Figure 1: Overview of the methodology for constructing domain-specific knowledge graphs and evaluating their impact on RAG-enhanced healthcare LLMs. The pipeline includes abstract selection, knowledge graph construction, probe generation, and systematic evaluation across multiple models and retrieval configurations. Detailed explanation is provided in Section \ref{['sec:methodology']}.
• Figure 2: Trends across RAG configurations for the averaged F$_1$ scores of the models in Table \ref{['tab:probe1']} and \ref{['tab:probe2']} (averaged over $T=\{0.0,0.2,0.5\}$).