Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Tokenization for Clinical Time Series: When Less is More

Rafi Al Attrach, Rajna Fani, David Restrepo, Yugang Jia, Peter Schüffler

TL;DR

This study question how best to tokenize clinical time series for transformer models and whether common enhancements to time and value encoding actually help. Through a controlled, multi-task evaluation on MEDS-MIMIC-IV data, it shows that explicit time encodings often offer little advantage, while code-into-language-model representations (TextCode) benefit greatly when the code encoder is frozen and large, especially in clinical-domain pretrained models. The findings reveal that code sequences carry substantial predictive signal, with frozen, larger clinical encoders delivering strong, parameter-efficient performance across tasks. The work provides a fair evaluation framework for tokenization strategies and suggests a shift toward simpler, more scalable approaches for clinical AI.

Abstract

Tokenization strategies shape how models process electronic health records, yet fair comparisons of their effectiveness remain limited. We present a systematic evaluation of tokenization approaches for clinical time series modeling using transformer-based architectures, revealing task-dependent and sometimes counterintuitive findings about temporal and value feature importance. Through controlled ablations across four clinical prediction tasks on MIMIC-IV, we demonstrate that explicit time encodings provide no consistent statistically significant benefit for the evaluated downstream tasks. Value features show task-dependent importance, affecting mortality prediction but not readmission, suggesting code sequences alone can carry sufficient predictive signal. We further show that frozen pretrained code encoders dramatically outperform their trainable counterparts while requiring dramatically fewer parameters. Larger clinical encoders provide consistent improvements across tasks, benefiting from frozen embeddings that eliminate computational overhead. Our controlled evaluation enables fairer tokenization comparisons and demonstrates that simpler, parameter-efficient approaches can, in many cases, achieve strong performance, though the optimal tokenization strategy remains task-dependent.

Rethinking Tokenization for Clinical Time Series: When Less is More

TL;DR

This study question how best to tokenize clinical time series for transformer models and whether common enhancements to time and value encoding actually help. Through a controlled, multi-task evaluation on MEDS-MIMIC-IV data, it shows that explicit time encodings often offer little advantage, while code-into-language-model representations (TextCode) benefit greatly when the code encoder is frozen and large, especially in clinical-domain pretrained models. The findings reveal that code sequences carry substantial predictive signal, with frozen, larger clinical encoders delivering strong, parameter-efficient performance across tasks. The work provides a fair evaluation framework for tokenization strategies and suggests a shift toward simpler, more scalable approaches for clinical AI.

Abstract

Tokenization strategies shape how models process electronic health records, yet fair comparisons of their effectiveness remain limited. We present a systematic evaluation of tokenization approaches for clinical time series modeling using transformer-based architectures, revealing task-dependent and sometimes counterintuitive findings about temporal and value feature importance. Through controlled ablations across four clinical prediction tasks on MIMIC-IV, we demonstrate that explicit time encodings provide no consistent statistically significant benefit for the evaluated downstream tasks. Value features show task-dependent importance, affecting mortality prediction but not readmission, suggesting code sequences alone can carry sufficient predictive signal. We further show that frozen pretrained code encoders dramatically outperform their trainable counterparts while requiring dramatically fewer parameters. Larger clinical encoders provide consistent improvements across tasks, benefiting from frozen embeddings that eliminate computational overhead. Our controlled evaluation enables fairer tokenization comparisons and demonstrates that simpler, parameter-efficient approaches can, in many cases, achieve strong performance, though the optimal tokenization strategy remains task-dependent.

Paper Structure

This paper contains 20 sections, 2 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Clinical Tasks and Dataset
  4. Tokenization Approaches
  5. Controlled Experimental Design
  6. Statistical Analysis
  7. Results
  8. Triplet component importance
  9. TextCode: mapping, freezing, and code encoders
  10. Learnable time encodings
  11. Conclusion
  12. Implementation Details
  13. TextCode Encoder Architecture
  14. Model Specifications
  15. Controlled Experimental Settings
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Triplet component ablation across four tasks. Error bars show $\pm$sd over $N=10$ seeds. Full numeric results are deferred to Appendix \ref{['tab:triplet_full']}.
  • Figure 2: Efficiency frontier for TextCode variants on post-discharge mortality. Bubble size reflects trainable parameters. Freezing the encoder offers a superior performance/cost trade-off.