From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP

TL;DR

From Word Sequences to Behavioral Sequences reframes NLP prediction for longitudinal data by treating documents as time-ordered verb al behavior from individuals rather than independent samples. It introduces target-aligned evaluation with cross-sectional and prospective splits, and decomposes performance into between-person and within-person components to separate stable differences from dynamic change. The study demonstrates that traditional document-level evaluation can mislead, and shows that incorporating history through sequence inputs and appropriate temporal inductive biases yields robust gains under ecologically valid settings, with model choices depending on the generalization target. The work argues for a broader methodological shift toward behavior-sequence paradigms in NLP, with important implications for dataset construction, modeling, and ethical governance in sensitive domains like mental health.

Abstract

While NLP typically treats documents as independent and unordered samples, in longitudinal studies, this assumption rarely holds: documents are nested within authors and ordered in time, forming person-indexed, time-ordered $\textit{behavioral sequences}$. Here, we demonstrate the need for and propose a longitudinal modeling and evaluation paradigm that consequently updates four parts of the NLP pipeline: (1) evaluation splits aligned to generalization over people ($\textit{cross-sectional}$) and/or time ($\textit{prospective}$); (2) accuracy metrics separating between-person differences from within-person dynamics; (3) sequence inputs to incorporate history by default; and (4) model internals that support different $\textit{coarseness}$ of latent state over histories (pooled summaries, explicit dynamics, or interaction-based models). We demonstrate the issues ensued by traditional pipeline and our proposed improvements on a dataset of 17k daily diary transcripts paired with PTSD symptom severity from 238 participants, finding that traditional document-level evaluation can yield substantially different and sometimes reversed conclusions compared to our ecologically valid modeling and evaluation. We tie our results to a broader discussion motivating a shift from word-sequence evaluation toward $\textit{behavior-sequence}$ paradigms for NLP.

Figures (8)

• Figure 1: Flattened metrics can mask whether models learn people or dynamics.Left: For three illustrative users from cross-sectional test set, predictions covary within individuals, but the between-person relationship of person-level means is reversed in direction; a pooled document-level fit conflates these effects, depicted by the flattened fit. Right: Decomposing performance into between-person and within-person MAE/$r$ across split regimes separates individual differences from temporal variation and clarifies what drives apparent "good" performance.
• Figure 2: Between- and within-person MAE vs. representation size. Error shows a U-shaped trend across regimes: cross-sectional and cross-sectional & prospective perform best with small representations (64--128), whereas prospective benefits from larger size ($\sim$512).
• Figure 3: Between- and within-person MAE vs. history length. Using longer history generally improves performance, but the best history--capacity trade-off depends on the regime: prospective benefits from longer context with larger representations ($\geq$512), while cross-sectional and cross-sectional & prospective improve primarily with longer context at smaller size ($\sim$64). Across splits, the average best size is 128.
• Figure 4: AR vs. BoE vs. Transformer across history length (128 dims per day). Between-person (top) and within-person (bottom) MAE as a function of history $h$. AR performs best for cross-sectional generalization, while modeling temporal interactions (Transformer) yields the largest gains for prospective generalization but performs worst for cross-sectional & prospective.
• Figure 5: Between- (top) and Within-Person (bottom) SMAPE as a function of hidden dimension size. Forecasting performance follows a U-shaped trend as a function of hidden dimension size of language across all three evaluation sets. While a typical model requires only 64 dimensions of language for best performance on Cross-sectional and Cross-sectional & Prospective test sets, it requires 512 dimensions in Prospective evaluation set. Based on the best performance achieved in different settings, generalization to Cross-sectional & Prospective is the hardest, followed by Cross-sectional set and finally prospective set. Generalization to unseen people is harder than unseen time and within-person changes is harder than between-person differences.