Comparing representations of long clinical texts for the task of patient note-identification

TL;DR

This study tackles patient-note identification by building patient representations from their clinical notes and evaluating multiple embedding schemes. It compares HAN BERTLSTM, HTN, Longformer, and BERT across token-embedding and [CLS] configurations, with and without sliding windows, and assesses pooling methods including the effective mean_max approach defined as $ \mathbf{R} = [\text{mean}(\mathbf{r}_1,\dots,\mathbf{r}_m) \oplus \text{max}(\mathbf{r}_1,\dots,\mathbf{r}_m)]$, using five classifiers and standard metrics on MIMIC-III and Necker data. The results show that BERT-based embeddings with a sliding window and mean_max pooling consistently yield the highest accuracy and AUC, and that this approach generalizes to French clinical notes via CamemBERT. The findings underscore the importance of embedding choice and aggregation strategy for robust patient-level modeling, with practical implications for deduplication, privacy, and similarity analyses in real-world EHR systems.

Abstract

In this paper, we address the challenge of patient-note identification, which involves accurately matching an anonymized clinical note to its corresponding patient, represented by a set of related notes. This task has broad applications, including duplicate records detection and patient similarity analysis, which require robust patient-level representations. We explore various embedding methods, including Hierarchical Attention Networks (HAN), three-level Hierarchical Transformer Networks (HTN), LongFormer, and advanced BERT-based models, focusing on their ability to process mediumto-long clinical texts effectively. Additionally, we evaluate different pooling strategies (mean, max, and mean_max) for aggregating wordlevel embeddings into patient-level representations and we examine the impact of sliding windows on model performance. Our results indicate that BERT-based embeddings outperform traditional and hierarchical models, particularly in processing lengthy clinical notes and capturing nuanced patient representations. Among the pooling strategies, mean_max pooling consistently yields the best results, highlighting its ability to capture critical features from clinical notes. Furthermore, the reproduction of our results on both MIMIC dataset and Necker hospital data warehouse illustrates the generalizability of these approaches to real-world applications, emphasizing the importance of both embedding methods and aggregation strategies in optimizing patient-note identification and enhancing patient-level modeling.

Figures (5)

• Figure 1: Cohort design, MIMIC-III dataset.
• Figure 2: Overview of the HAN architecture incorporating BERT and BiLSTM with attention or pooling strategies for hierarchical aggregation. Adapted from Si2021ThreelevelHT.
• Figure 3: Overview of the HTN architecture incorporating BERT and Multi-head Transformer Encoder with pooling strategies for hierarchical aggregation. Adapted from Si2021ThreelevelHT.
• Figure 4: Number of tokens across MIMIC-III Physician notes.
• Figure 5: Sliding window approach.