Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TEE4EHR: Transformer Event Encoder for Better Representation Learning in Electronic Health Records

Hojjat Karami, David Atienza, Anisoara Ionescu

TL;DR

The paper tackles irregular sampling and informative missingness in EHR time series by modeling laboratory-test patterns as events using a neural point process. It introduces TEE4EHR, combining a Transformer Event Encoder (TEE) with a Deep Attention Module (DAM), trained with point-process loss to learn conditional intensity functions. Experiments on benchmark event-sequence datasets and two real-world ICU datasets show that TEE+DAM improves future-event prediction and downstream clinical outcomes, while producing more interpretable patient embeddings via attention aggregation. These results highlight improved representation learning in EHRs and potential benefits for clinical decision support and patient phenotyping.

Abstract

Irregular sampling of time series in electronic health records (EHRs) is one of the main challenges for developing machine learning models. Additionally, the pattern of missing data in certain clinical variables is not at random but depends on the decisions of clinicians and the state of the patient. Point process is a mathematical framework for analyzing event sequence data that is consistent with irregular sampling patterns. Our model, TEE4EHR, is a transformer event encoder (TEE) with point process loss that encodes the pattern of laboratory tests in EHRs. The utility of our TEE has been investigated in a variety of benchmark event sequence datasets. Additionally, we conduct experiments on two real-world EHR databases to provide a more comprehensive evaluation of our model. Firstly, in a self-supervised learning approach, the TEE is jointly learned with an existing attention-based deep neural network which gives superior performance in negative log-likelihood and future event prediction. Besides, we propose an algorithm for aggregating attention weights that can reveal the interaction between the events. Secondly, we transfer and freeze the learned TEE to the downstream task for the outcome prediction, where it outperforms state-of-the-art models for handling irregularly sampled time series. Furthermore, our results demonstrate that our approach can improve representation learning in EHRs and can be useful for clinical prediction tasks.

TEE4EHR: Transformer Event Encoder for Better Representation Learning in Electronic Health Records

TL;DR

The paper tackles irregular sampling and informative missingness in EHR time series by modeling laboratory-test patterns as events using a neural point process. It introduces TEE4EHR, combining a Transformer Event Encoder (TEE) with a Deep Attention Module (DAM), trained with point-process loss to learn conditional intensity functions. Experiments on benchmark event-sequence datasets and two real-world ICU datasets show that TEE+DAM improves future-event prediction and downstream clinical outcomes, while producing more interpretable patient embeddings via attention aggregation. These results highlight improved representation learning in EHRs and potential benefits for clinical decision support and patient phenotyping.

Abstract

Irregular sampling of time series in electronic health records (EHRs) is one of the main challenges for developing machine learning models. Additionally, the pattern of missing data in certain clinical variables is not at random but depends on the decisions of clinicians and the state of the patient. Point process is a mathematical framework for analyzing event sequence data that is consistent with irregular sampling patterns. Our model, TEE4EHR, is a transformer event encoder (TEE) with point process loss that encodes the pattern of laboratory tests in EHRs. The utility of our TEE has been investigated in a variety of benchmark event sequence datasets. Additionally, we conduct experiments on two real-world EHR databases to provide a more comprehensive evaluation of our model. Firstly, in a self-supervised learning approach, the TEE is jointly learned with an existing attention-based deep neural network which gives superior performance in negative log-likelihood and future event prediction. Besides, we propose an algorithm for aggregating attention weights that can reveal the interaction between the events. Secondly, we transfer and freeze the learned TEE to the downstream task for the outcome prediction, where it outperforms state-of-the-art models for handling irregularly sampled time series. Furthermore, our results demonstrate that our approach can improve representation learning in EHRs and can be useful for clinical prediction tasks.
Paper Structure (25 sections, 16 equations, 3 figures, 6 tables)

This paper contains 25 sections, 16 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background & Related Works
  3. Problem Formulation
  4. Point Process Framework
  5. Deep learning for irregularly sampled data
  6. Proposed Model
  7. Transformer Event Encoder
  8. Deep Attention Module
  9. Event Decoder
  10. Experiments
  11. Datasets
  12. Preliminary evaluation of TEE
  13. TEE for EHRs
  14. Baselines
  15. Metrics
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Schematic representation of TEE4EHR model combining a transformer-based event encoder (TEE) with a deep attention module (DAM) that can handle irregularly sampled time series.
  • Figure 2: Attention aggregation in P12 dataset. (a) t-SNE plot of patients with the aggregated frequency matrix and aggregated interaction matrix for two subgroups of patients. (b) Top ten frequent measurement patterns for laboratory tests.
  • Figure 3: Pattern of similar patients in the embedding space. (a) An example patient in P19 dataset. (b) The four nearest neighbors in the embedding space that are learned by TEE+DAM (b), and DAM (c). Similar patterns are highlighted in yellow.