RelCAT: Advancing Extraction of Clinical Inter-Entity Relationships from Unstructured Electronic Health Records
Shubham Agarwal, Vlad Dinu, Thomas Searle, Mart Ratas, Anthony Shek, Dan F. Stein, James Teo, Richard Dobson
TL;DR
RelCAT introduces a complete toolkit to extract and classify inter-entity relations in clinical text by extending MedCAT within the CogStack framework. It evaluates transformer-based classifiers (BERT, Llama) and in-context learning with LLMs on gold-standard n2c2 data and real-world NHS datasets, achieving a macro F1 of 0.977 on n2c2—surpassing prior state-of-the-art—and strong NHS performance (>0.93 F1). The approach includes two workflows (end-to-end and standalone), robust handling of non-relations, and an annotation workflow via MedCATTrainer to build high-quality relation labels. The work demonstrates that supervised, fine-tuned models currently outperform zero-shot/few-shot LLM prompting for clinical relation extraction, while providing scalable paths for future ontology-based labeling and broader deployment in clinical analytics.
Abstract
This study introduces RelCAT (Relation Concept Annotation Toolkit), an interactive tool, library, and workflow designed to classify relations between entities extracted from clinical narratives. Building upon the CogStack MedCAT framework, RelCAT addresses the challenge of capturing complete clinical relations dispersed within text. The toolkit implements state-of-the-art machine learning models such as BERT and Llama along with proven evaluation and training methods. We demonstrate a dataset annotation tool (built within MedCATTrainer), model training, and evaluate our methodology on both openly available gold-standard and real-world UK National Health Service (NHS) hospital clinical datasets. We perform extensive experimentation and a comparative analysis of the various publicly available models with varied approaches selected for model fine-tuning. Finally, we achieve macro F1-scores of 0.977 on the gold-standard n2c2, surpassing the previous state-of-the-art performance, and achieve performance of >=0.93 F1 on our NHS gathered datasets.