A statistical perspective on transformers for small longitudinal cohort data

TL;DR

This work tackles the difficulty of applying transformer models to small longitudinal cohorts by introducing the MiniTransformer, a minimal, VAR-inspired transformer that uses a kernel-based, temporally decaying attention with scalar projections and a compact cumulant aggregation. A permutation-based test assesses context effects, enabling interpretable identification of how past observations contextualize future predictions. Empirical results from simulations and two resilience-study datasets show that the MiniTransformer achieves competitive predictive accuracy and reveals meaningful context patterns, even with limited sequences. Collectively, the approach demonstrates that carefully simplified transformer architectures can capture complex temporal dependencies in small, real-world biomedical data and offer interpretable insights for clinical interpretation.

Abstract

Modeling of longitudinal cohort data typically involves complex temporal dependencies between multiple variables. There, the transformer architecture, which has been highly successful in language and vision applications, allows us to account for the fact that the most recently observed time points in an individual's history may not always be the most important for the immediate future. This is achieved by assigning attention weights to observations of an individual based on a transformation of their values. One reason why these ideas have not yet been fully leveraged for longitudinal cohort data is that typically, large datasets are required. Therefore, we present a simplified transformer architecture that retains the core attention mechanism while reducing the number of parameters to be estimated, to be more suitable for small datasets with few time points. Guided by a statistical perspective on transformers, we use an autoregressive model as a starting point and incorporate attention as a kernel-based operation with temporal decay, where aggregation of multiple transformer heads, i.e. different candidate weighting schemes, is expressed as accumulating evidence on different types of underlying characteristics of individuals. This also enables a permutation-based statistical testing procedure for identifying contextual patterns. In a simulation study, the approach is shown to recover contextual dependencies even with a small number of individuals and time points. In an application to data from a resilience study, we identify temporal patterns in the dynamics of stress and mental health. This indicates that properly adapted transformers can not only achieve competitive predictive performance, but also uncover complex context dependencies in small data settings.

Figures (2)

• Figure 1: Individual simulated and fitted trajectories for a random selection of individuals, with true underlying structure $z_t$, observed target $x_{t,j_3}$, predicted target $\hat{x}_{t,j_3}$, and learned cumulants ( \ref{['eq:cumulant']}).
• Figure 2: Context-target effects for resilience dataset 1 (left) and dataset 2 (right). Dark blue indicates smaller values of the visualized test statistic $s^{(r)}_j$ (i.e., the context has a smaller effect), while dark red indicates larger values.