Exploring the Role of Knowledge Graph-Based RAG in Japanese Medical Question Answering with Small-Scale LLMs

TL;DR

The paper addresses Japanese medical question answering under privacy constraints that limit the use of large commercial LLMs. It proposes a knowledge graph-based retrieval-augmented generation framework that leverages the external knowledge base UMLS and word-level translation to support small open-source LLMs. Across three translated QA datasets and a range of models, KG-RAG shows only limited overall gains, with improvements highly sensitive to the relevance and quality of retrieved KG content. These findings highlight practical challenges and provide guidance for deploying RAG in low-resource languages, outlining avenues to enhance retrieval relevance and integration with biomedical knowledge sources.

Abstract

Large language models (LLMs) perform well in medical QA, but their effectiveness in Japanese contexts is limited due to privacy constraints that prevent the use of commercial models like GPT-4 in clinical settings. As a result, recent efforts focus on instruction-tuning open-source LLMs, though the potential of combining them with retrieval-augmented generation (RAG) remains underexplored. To bridge this gap, we are the first to explore a knowledge graph-based (KG) RAG framework for Japanese medical QA small-scale open-source LLMs. Experimental results show that KG-based RAG has only a limited impact on Japanese medical QA using small-scale open-source LLMs. Further case studies reveal that the effectiveness of the RAG is sensitive to the quality and relevance of the external retrieved content. These findings offer valuable insights into the challenges and potential of applying RAG in Japanese medical QA, while also serving as a reference for other low-resource languages.

Figures (4)

• Figure 1: The pipeline of our knowledge graph-based RAG mechanism. Given a Japanese medical question, relevant medical knowledge is retrieved from the external knowledge base and combined with the original Japanese question as input to the LLM for answer generation. The English translation and full content are provided in Appendix \ref{['appendix:pipeline']}.
• Figure 2: English translation and the full content of the question and answer in the Fig. \ref{['fig:pipeline']}
• Figure 3: Full content and English translation of the case 1 in Table \ref{['tab:casestudy']}.
• Figure 4: Full content and English translation of the case 2 in Table \ref{['tab:casestudy']}.