Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Document-level Clinical Entity and Relation Extraction via Knowledge Base-Guided Generation

Kriti Bhattarai, Inez Y. Oh, Zachary B. Abrams, Albert M. Lai

TL;DR

The paper addresses the challenge of high-precision clinical information extraction by integrating a medical knowledge base with large language models. It proposes a framework that maps UMLS concepts from clinical text using MetaMap, filters for medication-related concepts, and encodes them into dynamic prompts to guide GPT-based entity and relation extraction at the document level. Empirical results on the n2c2 and ADE datasets show that UMLS-guided prompts substantially improve F1 scores, particularly with GPT-4-32k, and outperform baseline few-shot and RAG approaches in many settings. The work demonstrates the practical value of combining structured domain knowledge with contextual language models to enhance healthcare text mining and suggests avenues for broader concept coverage and model diversification.

Abstract

Generative pre-trained transformer (GPT) models have shown promise in clinical entity and relation extraction tasks because of their precise extraction and contextual understanding capability. In this work, we further leverage the Unified Medical Language System (UMLS) knowledge base to accurately identify medical concepts and improve clinical entity and relation extraction at the document level. Our framework selects UMLS concepts relevant to the text and combines them with prompts to guide language models in extracting entities. Our experiments demonstrate that this initial concept mapping and the inclusion of these mapped concepts in the prompts improves extraction results compared to few-shot extraction tasks on generic language models that do not leverage UMLS. Further, our results show that this approach is more effective than the standard Retrieval Augmented Generation (RAG) technique, where retrieved data is compared with prompt embeddings to generate results. Overall, we find that integrating UMLS concepts with GPT models significantly improves entity and relation identification, outperforming the baseline and RAG models. By combining the precise concept mapping capability of knowledge-based approaches like UMLS with the contextual understanding capability of GPT, our method highlights the potential of these approaches in specialized domains like healthcare.

Document-level Clinical Entity and Relation Extraction via Knowledge Base-Guided Generation

TL;DR

The paper addresses the challenge of high-precision clinical information extraction by integrating a medical knowledge base with large language models. It proposes a framework that maps UMLS concepts from clinical text using MetaMap, filters for medication-related concepts, and encodes them into dynamic prompts to guide GPT-based entity and relation extraction at the document level. Empirical results on the n2c2 and ADE datasets show that UMLS-guided prompts substantially improve F1 scores, particularly with GPT-4-32k, and outperform baseline few-shot and RAG approaches in many settings. The work demonstrates the practical value of combining structured domain knowledge with contextual language models to enhance healthcare text mining and suggests avenues for broader concept coverage and model diversification.

Abstract

Generative pre-trained transformer (GPT) models have shown promise in clinical entity and relation extraction tasks because of their precise extraction and contextual understanding capability. In this work, we further leverage the Unified Medical Language System (UMLS) knowledge base to accurately identify medical concepts and improve clinical entity and relation extraction at the document level. Our framework selects UMLS concepts relevant to the text and combines them with prompts to guide language models in extracting entities. Our experiments demonstrate that this initial concept mapping and the inclusion of these mapped concepts in the prompts improves extraction results compared to few-shot extraction tasks on generic language models that do not leverage UMLS. Further, our results show that this approach is more effective than the standard Retrieval Augmented Generation (RAG) technique, where retrieved data is compared with prompt embeddings to generate results. Overall, we find that integrating UMLS concepts with GPT models significantly improves entity and relation identification, outperforming the baseline and RAG models. By combining the precise concept mapping capability of knowledge-based approaches like UMLS with the contextual understanding capability of GPT, our method highlights the potential of these approaches in specialized domains like healthcare.
Paper Structure (27 sections, 4 figures, 4 tables)

This paper contains 27 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Few-shot in-context learning
  4. Knowledge base-guided models
  5. Methods
  6. Overview of the Framework
  7. UMLS Integration in Large Language Model
  8. Datasets Description
  9. Results
  10. Experimental Setup
  11. Dataset
  12. Performance Results
  13. Discussion and Conclusion
  14. Limitations and Future Work
  15. Ethics Statement
  16. ...and 12 more sections

Figures (4)

  • Figure 1: (A) Step-by-step approach to integrating UMLS and extracting relation pairs. (B) Example of UMLS concepts mapped from the text. Some of the concepts, such as Prednisone, are recognized by GPT, as they are concepts GPT model is inherently trained on. However, concepts such as ASA, Cipro, Plavix are not recognized by GPT; UMLS facilitates their recognition.
  • Figure 2: An example of a prompt used to extract dosage information from the text using the UMLS concepts. The ‘note_text’ represents each text instance from ADE or n2c2 corpus. The ‘medication_list’ represents the UMLS concepts extracted from MetaMap.
  • Figure 3: Sample text of discharge summaries in the (A) n2c2 dataset and (B) ADE Corpus. The text highlighted in red are the targeted entities for extraction
  • Figure :