Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Clinical Knowledge Graph Construction and Evaluation with Multi-LLMs via Retrieval-Augmented Generation

Udiptaman Das, Krishnasai B. Atmakuri, Duy Ho, Chi Lee, Yugyung Lee

TL;DR

This work presents KG-RAG, an end-to-end framework for building and validating clinical knowledge graphs directly from unstructured oncology narratives. It employs a multi-agent prompting cascade with Gemini 2.0 Flash for schema-guided EAV extraction, GPT-4o for enrichment and reflection-based refinement, and Grok 3 for adversarial validation, all grounded to ontologies such as SNOMED CT, LOINC, RxNorm, GO, and ICD encoded in RDF/RDFS/OWL. The methodology spans five stages—EAV extraction, ontology mapping, relation discovery, semantic encoding, and trust-based validation—producing SPARQL-compatible graphs with composite trust scores. The approach demonstrates robust factual grounding, semantic coherence, and ontology compliance across PDAC and BRCA cohorts in CORAL, enabling explainable clinical AI and decision support with continuous self-supervision and refinement.

Abstract

Large language models (LLMs) offer new opportunities for constructing knowledge graphs (KGs) from unstructured clinical narratives. However, existing approaches often rely on structured inputs and lack robust validation of factual accuracy and semantic consistency, limitations that are especially problematic in oncology. We introduce an end-to-end framework for clinical KG construction and evaluation directly from free text using multi-agent prompting and a schema-constrained Retrieval-Augmented Generation (KG-RAG) strategy. Our pipeline integrates (1) prompt-driven entity, attribute, and relation extraction; (2) entropy-based uncertainty scoring; (3) ontology-aligned RDF/OWL schema generation; and (4) multi-LLM consensus validation for hallucination detection and semantic refinement. Beyond static graph construction, the framework supports continuous refinement and self-supervised evaluation, enabling iterative improvement of graph quality. Applied to two oncology cohorts (PDAC and BRCA), our method produces interpretable, SPARQL-compatible, and clinically grounded knowledge graphs without relying on gold-standard annotations. Experimental results demonstrate consistent gains in precision, relevance, and ontology compliance over baseline methods.

Clinical Knowledge Graph Construction and Evaluation with Multi-LLMs via Retrieval-Augmented Generation

TL;DR

This work presents KG-RAG, an end-to-end framework for building and validating clinical knowledge graphs directly from unstructured oncology narratives. It employs a multi-agent prompting cascade with Gemini 2.0 Flash for schema-guided EAV extraction, GPT-4o for enrichment and reflection-based refinement, and Grok 3 for adversarial validation, all grounded to ontologies such as SNOMED CT, LOINC, RxNorm, GO, and ICD encoded in RDF/RDFS/OWL. The methodology spans five stages—EAV extraction, ontology mapping, relation discovery, semantic encoding, and trust-based validation—producing SPARQL-compatible graphs with composite trust scores. The approach demonstrates robust factual grounding, semantic coherence, and ontology compliance across PDAC and BRCA cohorts in CORAL, enabling explainable clinical AI and decision support with continuous self-supervision and refinement.

Abstract

Large language models (LLMs) offer new opportunities for constructing knowledge graphs (KGs) from unstructured clinical narratives. However, existing approaches often rely on structured inputs and lack robust validation of factual accuracy and semantic consistency, limitations that are especially problematic in oncology. We introduce an end-to-end framework for clinical KG construction and evaluation directly from free text using multi-agent prompting and a schema-constrained Retrieval-Augmented Generation (KG-RAG) strategy. Our pipeline integrates (1) prompt-driven entity, attribute, and relation extraction; (2) entropy-based uncertainty scoring; (3) ontology-aligned RDF/OWL schema generation; and (4) multi-LLM consensus validation for hallucination detection and semantic refinement. Beyond static graph construction, the framework supports continuous refinement and self-supervised evaluation, enabling iterative improvement of graph quality. Applied to two oncology cohorts (PDAC and BRCA), our method produces interpretable, SPARQL-compatible, and clinically grounded knowledge graphs without relying on gold-standard annotations. Experimental results demonstrate consistent gains in precision, relevance, and ontology compliance over baseline methods.
Paper Structure (28 sections, 13 equations, 4 figures, 5 tables)

This paper contains 28 sections, 13 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Overview of Multi-Agent KG Construction and Evaluation Pipeline
  5. Stage 1: FHIR-Guided EAV Extraction
  6. Triple Extraction.
  7. FHIR Typing Function.
  8. Entropy-Based Value Confidence.
  9. Illustrative EAV Triples.
  10. Stage 2: Ontology Mapping & Schema Construction
  11. Ontology Vocabulary.
  12. Concept Mapping.
  13. Schema Construction.
  14. Persistent URIs.
  15. Outcome.
  16. ...and 13 more sections

Figures (4)

  • Figure 1: PDAC vs. BRCA Knowledge Graph Metrics by EAV Statistics, Ontology Mapping, Predicate Typing, and Graph Structure. BRCA emphasizes molecular attributes; PDAC reflects procedural diversity.
  • Figure 2: Comparison of attribute-level metrics for PDAC and BRCA cohorts. From left to right: raw coverage percentage, correctness percentage, incorrect attribute counts, and overall averages.
  • Figure 3: Top: Per-patient factual correctness of EAV triples across three LLMs. Bottom: Pairwise correctness differences and Pearson correlation trends across 40 patients.
  • Figure 4: Radar chart comparison of Gemini 2.0 Flash, Grok 3, and GPT-4o: (a) relation diversity and structural metrics, (b) semantic issue categories, and (c) unified evaluation across correctness, inference, and data support.