Diagnosis extraction from unstructured Dutch echocardiogram reports using span- and document-level characteristic classification

TL;DR

This study tackles automatic extraction of diagnoses from unstructured Dutch echocardiogram reports to enable scalable labeling for clinical ML. It systematically compares span- and document-level NLP methods across eleven cardiac characteristics, finding that SpanCategorizer excels at span labeling while MedRoBERTa.nl dominates document labeling; SetFit provides a strong low-data option. Reported results include span-level weighted F1 scores ranging from $0.60$ to $0.93$ and document-level weighted F1 scores exceeding $0.96$ for all characteristics, with further gains when reducing label complexity. The published SpanCategorizer and MedRoBERTa.nl models support practical deployment for cohort construction and automated labeling, and future work includes external validation, extension to other cardiac features, and exploring cross-validation and joint extraction approaches.

Abstract

Clinical machine learning research and AI driven clinical decision support models rely on clinically accurate labels. Manually extracting these labels with the help of clinical specialists is often time-consuming and expensive. This study tests the feasibility of automatic span- and document-level diagnosis extraction from unstructured Dutch echocardiogram reports. We included 115,692 unstructured echocardiogram reports from the UMCU a large university hospital in the Netherlands. A randomly selected subset was manually annotated for the occurrence and severity of eleven commonly described cardiac characteristics. We developed and tested several automatic labelling techniques at both span and document levels, using weighted and macro F1-score, precision, and recall for performance evaluation. We compared the performance of span labelling against document labelling methods, which included both direct document classifiers and indirect document classifiers that rely on span classification results. The SpanCategorizer and MedRoBERTa$.$nl models outperformed all other span and document classifiers, respectively. The weighted F1-score varied between characteristics, ranging from 0.60 to 0.93 in SpanCategorizer and 0.96 to 0.98 in MedRoBERTa$.$nl. Direct document classification was superior to indirect document classification using span classifiers. SetFit achieved competitive document classification performance using only 10% of the training data. Utilizing a reduced label set yielded near-perfect document classification results. We recommend using our published SpanCategorizer and MedRoBERTa$.$nl models for span- and document-level diagnosis extraction from Dutch echocardiography reports. For settings with limited training data, SetFit may be a promising alternative for document classification.

Figures (5)

• Figure 1: Example report with manual annotations. For presentation purposes, text has been translated to English.
• Figure 2: MedCAT pipeline for identifying and classifying medical concepts
• Figure 3: SpanCat pipeline for iterating and classifying n-gram spans using scanning windows of $1$-$25$ tokens
• Figure 4: BOW pipeline involving tokenization, tf-idf weighting, topic modelling, and classification using a gradient-boosted classifier
• Figure 5: SetFit pipeline: fine-tuning the sentence encoder with label-based contrastive learning, followed by classification