Comparison of biomedical relationship extraction methods and models for knowledge graph creation

TL;DR

This work addresses the scalable extraction of normalized biomedical relationships to populate knowledge graphs from the literature. It systematically compares rule-based, traditional ML, and transformer-based approaches (DistilBERT, PubMedBERT, T5, SciFive) across drug–gene, drug–disease, and gene–disease relations, including mode of action and negation/speculation. The authors propose a detailed relationship model and an in-house Linnaeus-based NER/NEN pipeline, and they build large-scale graphs from PubMed with evidence sentences. Key findings show transformer-based models, particularly PubMedBERT, achieve high F1-scores (up to $0.92$) on balanced data, with DistilBERT offering a favorable balance of speed and performance; rule-based methods remain valuable for precision and baseline graph construction, while domain-specific variants provide modest gains. The results have practical impact for drug discovery and pharmacovigilance by enabling richer, evidence-backed knowledge graphs that support multi-hop reasoning and hypothesis generation.

Abstract

Biomedical research is growing at such an exponential pace that scientists, researchers, and practitioners are no more able to cope with the amount of published literature in the domain. The knowledge presented in the literature needs to be systematized in such a way that claims and hypotheses can be easily found, accessed, and validated. Knowledge graphs can provide such a framework for semantic knowledge representation from literature. However, in order to build a knowledge graph, it is necessary to extract knowledge as relationships between biomedical entities and normalize both entities and relationship types. In this paper, we present and compare few rule-based and machine learning-based (Naive Bayes, Random Forests as examples of traditional machine learning methods and DistilBERT, PubMedBERT, T5 and SciFive-based models as examples of modern deep learning transformers) methods for scalable relationship extraction from biomedical literature, and for the integration into the knowledge graphs. We examine how resilient are these various methods to unbalanced and fairly small datasets. Our experiments show that transformer-based models handle well both small (due to pre-training on a large dataset) and unbalanced datasets. The best performing model was the PubMedBERT-based model fine-tuned on balanced data, with a reported F1-score of 0.92. DistilBERT-based model followed with F1-score of 0.89, performing faster and with lower resource requirements. BERT-based models performed better then T5-based generative models.

Figures (3)

• Figure 1: Example of dictionaries, rules set and an example of sentence annotations in order to match relationship in a sentence
• Figure 2: Section of knowledge graph showing nodes that are in relationship with autosomal dominant polycystic kidney disease (ADPKD). Orange entities are diseases (ADPKD), entities in blue are drugs and in green are genes/proteins. Label on edges present relationship type, number of mentions and cumulative confidence score for the given relationship between two entities.
• Figure 3: F1-score by epoch in fine-tuned DistilBERT and T5 models on both unbalanced and balanced datasets