Automatic identification of diagnosis from hospital discharge letters via weakly-supervised Natural Language Processing

TL;DR

This paper tackles the challenge of identifying diagnoses from Italian hospital discharge letters without manual annotation. It introduces a weakly-supervised NLP pipeline combining domain-adapted transformer pre-training, diagnosis-string extraction with two-level clustering, and keyword-guided mapping to generate weak labels, which train a final classifier. On a bronchiolitis case study with 33,176 letters, the approach achieves competitive AUC and F1 scores close to supervised methods while saving substantial expert time, and it demonstrates robustness to weak-label choice and generalizability across diseases. The work offers a scalable framework for clinical text classification in low-resource languages and highlights the value of domain-specific pre-training and clustering-based weak-label generation for large, unlabelled datasets.

Abstract

Identifying patient diagnoses from discharge letters is essential to enable large-scale cohort selection and epidemiological research, but traditional supervised approaches rely on extensive manual annotation, which is often impractical for large textual datasets. In this study, we present a novel weakly-supervised Natural Language Processing pipeline designed to classify Italian discharge letters without requiring manual labelling. After extracting diagnosis-related sentences, the method leverages a transformer-based model with an additional pre-training on Italian medical documents to generate semantic embeddings. A two-level clustering procedure is applied to these embeddings, and the resulting clusters are mapped to the diseases of interest to derive weak labels for a subset of data, eventually used to train a transformer-based classifier. We evaluate the approach on a real-world case study on bronchiolitis in a corpus of 33,176 Italian discharge letters of children admitted to 44 emergency rooms or hospitals in the Veneto Region between 2017 and 2020. The pipeline achieves an area under the curve (AUC) of 77.68% ($\pm 4.30\%)$ and an F1-score of 78.14% ($\pm 4.89\%$) against manual annotations. Its performance surpasses other unsupervised methods and approaches fully supervised models, maintaining robustness to cluster selection and promising generalizability across different disease types. It allows saving approximately 3 minutes of expert time per discharge letter, resulting in more than 1,500 hours for a dataset like ours. This study demonstrates the feasibility of a weakly-supervised strategy for identifying diagnoses from Italian discharge letters. The pipeline achieves strong performance, is adaptable to various diseases, and offers a scalable solution for clinical text classification, reducing the need for manual annotation while maintaining reliable accuracy.

Figures (7)

• Figure 1: a. Diagram of the full pipeline for weakly-supervised classification of discharge letters. Step 0 (TPT) consists in additional pre-training of a transformer-based model on a dataset of medical documents in Italian, including documents auomatically translated into Italian. Step 1 (SAL) is the core of the pipeline, consisting in the automatic extraction of diagnosis strings from the letters, their clustering (exploiting the model from TPT) and the mapping of the clusters to the disease labels. This procedure leads to a weakly-labelled subset of data that is used in Step 2 (WLC) to train a weakly-supervised classificaton model. b. Detailed diagram of the clustering step for diagnosis strings, corresponding to Step 1.2 of the full pipeline. The extracted diagnoses are embedded with a transformer-based model, and, after a dimensionality reduction step, clustered. Clusters are summarized with relevant keywords and the clustering is repeated on these clusters' representations.
• Figure 2: AUC with varying percentages of weak labels. Mean AUC ($\pm95\%$ CI) on gold labels, with and without diagnosis strings, as different proportions of gold labels are replaced by weak labels. Results are averaged over 5 repetitions of 10-fold cross-validation.
• Figure A.1: A. Distribution of bronchiolitis cases among hospitals. B. Distribution of bronchiolitis cases among LHUs.
• Figure A.2: A. Distribution of records among hospitals (only hospitals with $> 50$ records). B. Distribution of records among LHUs (only LHUs with $> 50$ cases). For each hospital/LHU, the blue bar indicates the total number of records, the light blue bar the records from pediatric ER/departments and the value on top of the bars is the % of pediatric records in the hospital/LHU.
• Figure B.1: AUC ($\pm 95\%$ C.I.) on gold labels with and without pre-processing