Temporal fine-tuning for early risk detection

TL;DR

This work tackles Early Risk Detection (ERD) on the Web, where timely and accurate identification of at-risk users is required as posts accumulate. It introduces temporal fine-tuning, a combined single-objective approach that injects explicit time information into transformer inputs and optimizes a differentiable surrogate for the temporal ERD metric $ERDE_\theta$. The method augments inputs with a [TIME] token, organizes training/validation around post windows, and uses a loss that jointly accounts for precision and latency, enabling end-to-end optimization. Experiments on the Spanish MentalRiskES 2023 depression and eating disorders tasks show competitive results against top teams, demonstrating improved decision speed and maintained accuracy. The work highlights potential extensions to cross-language evaluation, novel temporal losses, and deeper analysis of temporal representations for interpretability and broader ERD applicability.

Abstract

Early Risk Detection (ERD) on the Web aims to identify promptly users facing social and health issues. Users are analyzed post-by-post, and it is necessary to guarantee correct and quick answers, which is particularly challenging in critical scenarios. ERD involves optimizing classification precision and minimizing detection delay. Standard classification metrics may not suffice, resorting to specific metrics such as ERDE(theta) that explicitly consider precision and delay. The current research focuses on applying a multi-objective approach, prioritizing classification performance and establishing a separate criterion for decision time. In this work, we propose a completely different strategy, temporal fine-tuning, which allows tuning transformer-based models by explicitly incorporating time within the learning process. Our method allows us to analyze complete user post histories, tune models considering different contexts, and evaluate training performance using temporal metrics. We evaluated our proposal in the depression and eating disorders tasks for the Spanish language, achieving competitive results compared to the best models of MentalRiskES 2023. We found that temporal fine-tuning optimized decisions considering context and time progress. In this way, by properly taking advantage of the power of transformers, it is possible to address ERD by combining precision and speed as a single objective.

Figures (5)

• Figure 1: Pipeline of the temporal fine-tuning process. The samples are modified including time, temporal fine-tuning is applied according to an ERDE$\theta$, and the best model is chosen to be evaluated in an ERD environment.
• Figure 2: Temporal fine-tuning scheme for an epoch. The training and validation stages are subdivided into delays, where users are evaluated according to post windows. In training, loss is calculated at the end of each delay considering CrossEntropy and $lc_\theta$. In validation, the model performance is evaluated considering loss, accuracy, and ERDE$\theta$.
• Figure 3: Model learning during temporal fine-tuning for depression task. (a) Training stage; (b) Some cases of (a) detected in epoch:0 and corrected in epoch:1; (c) Validation stage. On the x-axis, the vertical bars depict the model’s decisions, and their length shows the number of posts read and the instance of the final response. Green bar: correct decision; red bar: wrong decision; gray bar: unread posts during analysis. The y-axis shows delays every 10 posts. The dashed horizontal line denotes the limit at $\theta$=30.
• Figure 4: Evaluation of the models using temporal fine-tuning validation and mock-server for both tasks.
• Figure 5: Temporal representation of a sentence evaluated at times t=[10, 20, …, 100]. Green dot: positive decision; red dot: negative decision. Embeddings were extracted from the fine-tuned model ([CLS] token), followed by dimensionality reduction through Principal Component Analysis (PCA).