METHOD: Modular Efficient Transformer for Health Outcome Discovery

TL;DR

METHOD addresses the challenges of modelling irregular clinical timelines with a specialized transformer that combines a patient-aware attention mechanism, adaptive sliding window attention, and a U‑Net–style long-sequence processor. It extends ETHOS by mitigating cross-patient information leakage, enabling multi-scale temporal learning, and preserving clinical hierarchies through dynamic skip connections, achieving superior performance on MIMIC‑IV especially for high-severity SOFA predictions. The work introduces a comprehensive evaluation framework spanning continuous and token-level metrics and reveals that METHOD maintains stable performance across varying history lengths while producing more clinically meaningful ICD embeddings. Collectively, METHOD represents a significant step toward clinically valid, efficient transformer models for healthcare data, with potential impact on real-world decision support and patient outcomes.

Abstract

Recent advances in transformer architectures have revolutionised natural language processing, but their application to healthcare domains presents unique challenges. Patient timelines are characterised by irregular sampling, variable temporal dependencies, and complex contextual relationships that differ substantially from traditional language tasks. This paper introduces \METHOD~(Modular Efficient Transformer for Health Outcome Discovery), a novel transformer architecture specifically designed to address the challenges of clinical sequence modelling in electronic health records. \METHOD~integrates three key innovations: (1) a patient-aware attention mechanism that prevents information leakage whilst enabling efficient batch processing; (2) an adaptive sliding window attention scheme that captures multi-scale temporal dependencies; and (3) a U-Net inspired architecture with dynamic skip connections for effective long sequence processing. Evaluations on the MIMIC-IV database demonstrate that \METHOD~consistently outperforms the state-of-the-art \ETHOS~model, particularly in predicting high-severity cases that require urgent clinical intervention. \METHOD~exhibits stable performance across varying inference lengths, a crucial feature for clinical deployment where patient histories vary significantly in length. Analysis of learned embeddings reveals that \METHOD~better preserves clinical hierarchies and relationships between medical concepts. These results suggest that \METHOD~represents a significant advancement in transformer architectures optimised for healthcare applications, providing more accurate and clinically relevant predictions whilst maintaining computational efficiency.

Figures (12)

• Figure 1: Comparison of attention masking strategies: (a) Standard causal mask enforces a strictly autoregressive structure where tokens can only attend to previous positions. (b) Our proposed patient-aware mask preserves causal dependencies within each patient sequence while preventing cross-patient information leakage and allowing global access to static patient context.
• Figure 2: Advanced attention masking in Method: (a) Sliding window mask that restricts attention computation to a local context window, enabling efficient processing of long sequences. (b) The combined Method mask integrates patient-aware block masking with sliding window attention to achieve both information isolation and efficient long-range dependency modelling.
• Figure 3: Performance metrics across different training sequence lengths. (a) Continuous SOFA MAE shows initial improvement followed by stabilisation. (b) Token-level MAE demonstrates consistent improvement with longer sequences. (c) Macro AUC indicates the optimal performance of around 3072 tokens. Error bars indicate the standard deviation over 10 runs.
• Figure 4: Performance heatmap across different inference lengths for the model trained with 32768 sequence length. Darker colours indicate better performance. The relatively uniform colouring across inference lengths suggests stable performance regardless of inference sequence length.
• Figure 5: Comparison of models trained with different sequence lengths (16384 vs 32768) across various inference lengths. (a) Continuous MAE shows consistent performance across inference lengths. (b) Token MAE demonstrates the stability of predictions regardless of inference length. (c) Macro AUC indicates robust discriminative ability across different sequence lengths.