Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can LLMs Support Medical Knowledge Imputation? An Evaluation-Based Perspective

Xinyu Yao, Aditya Sannabhadti, Holly Wiberg, Karmel S. Shehadeh, Rema Padman

TL;DR

This work addresses the problem of incomplete medical knowledge graphs, focusing on treatment mappings and the potential of Large Language Models (LLMs) to impute missing disease–treatment links. It introduces a knowledge-grounded evaluation framework that compares LLM-generated relationships against a curated reference KG constructed from UMLS, MONDO, ATC, ICD9-CM, DrugBank, RepoDB, and PrimeKG. The study finds that LLMs can surface plausible treatment candidates but exhibit hallucinations, cross-model inconsistencies, and limited recall, highlighting the need for grounding and hybrid validation—such as retrieval-augmented approaches—for safe integration into clinical decision support. Overall, the work provides a scalable evaluation methodology and practical guidance for deploying LLMs in medical knowledge augmentation while preserving safety and interpretability.

Abstract

Medical knowledge graphs (KGs) are essential for clinical decision support and biomedical research, yet they often exhibit incompleteness due to knowledge gaps and structural limitations in medical coding systems. This issue is particularly evident in treatment mapping, where coding systems such as ICD, Mondo, and ATC lack comprehensive coverage, resulting in missing or inconsistent associations between diseases and their potential treatments. To address this issue, we have explored the use of Large Language Models (LLMs) for imputing missing treatment relationships. Although LLMs offer promising capabilities in knowledge augmentation, their application in medical knowledge imputation presents significant risks, including factual inaccuracies, hallucinated associations, and instability between and within LLMs. In this study, we systematically evaluate LLM-driven treatment mapping, assessing its reliability through benchmark comparisons. Our findings highlight critical limitations, including inconsistencies with established clinical guidelines and potential risks to patient safety. This study serves as a cautionary guide for researchers and practitioners, underscoring the importance of critical evaluation and hybrid approaches when leveraging LLMs to enhance treatment mappings on medical knowledge graphs.

Can LLMs Support Medical Knowledge Imputation? An Evaluation-Based Perspective

TL;DR

This work addresses the problem of incomplete medical knowledge graphs, focusing on treatment mappings and the potential of Large Language Models (LLMs) to impute missing disease–treatment links. It introduces a knowledge-grounded evaluation framework that compares LLM-generated relationships against a curated reference KG constructed from UMLS, MONDO, ATC, ICD9-CM, DrugBank, RepoDB, and PrimeKG. The study finds that LLMs can surface plausible treatment candidates but exhibit hallucinations, cross-model inconsistencies, and limited recall, highlighting the need for grounding and hybrid validation—such as retrieval-augmented approaches—for safe integration into clinical decision support. Overall, the work provides a scalable evaluation methodology and practical guidance for deploying LLMs in medical knowledge augmentation while preserving safety and interpretability.

Abstract

Medical knowledge graphs (KGs) are essential for clinical decision support and biomedical research, yet they often exhibit incompleteness due to knowledge gaps and structural limitations in medical coding systems. This issue is particularly evident in treatment mapping, where coding systems such as ICD, Mondo, and ATC lack comprehensive coverage, resulting in missing or inconsistent associations between diseases and their potential treatments. To address this issue, we have explored the use of Large Language Models (LLMs) for imputing missing treatment relationships. Although LLMs offer promising capabilities in knowledge augmentation, their application in medical knowledge imputation presents significant risks, including factual inaccuracies, hallucinated associations, and instability between and within LLMs. In this study, we systematically evaluate LLM-driven treatment mapping, assessing its reliability through benchmark comparisons. Our findings highlight critical limitations, including inconsistencies with established clinical guidelines and potential risks to patient safety. This study serves as a cautionary guide for researchers and practitioners, underscoring the importance of critical evaluation and hybrid approaches when leveraging LLMs to enhance treatment mappings on medical knowledge graphs.

Paper Structure

This paper contains 15 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Literature
  3. Traditional Approaches to Knowledge Graph Completion
  4. Large Language Models for Biomedical Knowledge Augmentation
  5. Evaluating LLM-Generated Medical Knowledge
  6. Data and Methods
  7. Reference Knowledge Graph Construction
  8. LLM Querying and Prompt Design
  9. Evaluation Types and Metrics
  10. Results
  11. Coverage Performance of LLMs against KG reference
  12. Alignment Performance Across LLMs
  13. Robustness Check
  14. Conclusion and Future Work
  15. Future work

Figures (3)

  • Figure 1: Coverage Performance of LLMs against KG Reference List for Three Prompts
  • Figure 2: Alignment Performance of Accross LLMs and KG Reference List for Three Prompts
  • Figure 3: Consistency and Robustness of Perplexity-Sonar Across Multiple Iterations with API and Web Chat Portal