Benchmarking Probabilistic Time Series Forecasting Models on Neural Activity

TL;DR

This work addresses forecasting neural activity with probabilistic, uncertainty-aware predictions by benchmarking eight probabilistic deep learning models (including two foundation models) and four classical baselines on spontaneous widefield calcium imaging data from mouse cortex. It finds that deep models such as PatchTST, TiDE, and fine-tuned Chronos consistently outperform baselines across multiple forecast horizons, while zero-shot transfers for Chronos and Moirai underperform until fine-tuned. Informative forecasts extend up to about $1.5$ seconds into the future, with reliability highest near $1$ second ($\approx 35$ steps) and diminishing beyond $1.5$ seconds, as uncertainty grows and predictions resemble the training distribution mean. These results motivate the development of neuroscience-focused forecasting foundations and underscore the value of uncertainty quantification for potential closed-loop neural control applications, while also highlighting intrinsic neural variability as a limiting factor.

Abstract

Neural activity forecasting is central to understanding neural systems and enabling closed-loop control. While deep learning has recently advanced the state-of-the-art in the time series forecasting literature, its application to neural activity forecasting remains limited. To bridge this gap, we systematically evaluated eight probabilistic deep learning models, including two foundation models, that have demonstrated strong performance on general forecasting benchmarks. We compared them against four classical statistical models and two baseline methods on spontaneous neural activity recorded from mouse cortex via widefield imaging. Across prediction horizons, several deep learning models consistently outperformed classical approaches, with the best model producing informative forecasts up to 1.5 seconds into the future. Our findings point toward future control applications and open new avenues for probing the intrinsic temporal structure of neural activity.

Figures (4)

• Figure 1: a. Widefield imaging data aligned to Allen CCF, with example activity traces from one session. b. True and predicted activity in three test samples; dashed lines mark forecast start. Shaded regions: prediction intervals (P.I.): dark = 20%, light = 60%. Top: PatchTST and AR perform similarly, both outperforming Naive. Middle: PatchTST outperforms both AR and Naive. Bottom: PatchTST and AR perform comparably to Naive. Quantitative performance in each sample is listed on the right. c. Model performance at three forecast horizons. Models sorted by mean performance across sessions. The y-axis indicates relative performance, computed as the Mean Weighted Quantile Loss (MWQL, see Appendix\ref{['Appendix: models']} for definition) across all predicted steps of each model divided by that of the Naive baseline (i.e., y=1 corresponds to Naive performance). Colors indicate different sessions. Errorbar: mean $\pm$ std across 5 random seeds; fine-tuning Chronos yielded identical results across random seeds, likely due to an issue in the AutoGluon implementation. Performances of Theta and Average are omitted, as they are close to Naive and worse than all other models shown; for 70-step forecast, Average outperforms Naive but still not the others. AR(aicc) vs. AR(valQL): order chosen by AICC vs. validation MWQL. Models with stars significantly outperform AR(valQL) (one-sided paired t-test, $p < 0.05$). d. MWQL by forecast step. Errorbar: mean $\pm$ std across sessions. Bars and stars indicate steps where PatchTST significantly outperforms Naive and AR, respectively (one-sided paired t-test, $p < 0.05$). PatchTST and AR losses exceed 95% of the Average model loss (solid line) after 1.80s and 1.28s, respectively. e. Ratio of predicted distribution standard deviation (std) to training data std across forecast steps. Errorbar: mean $\pm$ std across sessions. Stars indicate steps where the PatchTST ratio is significantly less than 1 (one-sided t-test, $p<0.05$). In b, d, e, AR(valQL) is used as AR. For PatchTST, in each session the model achieving median MWQL across 5 random seeds is used.
• Figure 2: Evaluation on additional metrics. Models sorted by mean performance across sessions. Models in boldface significantly outperform AR(valQL) (one-sided paired t-test, $p < 0.05$). For Chronos and Moirai, -FT indicates the finetuned version, -ZS indicates zeroshot version.
• Figure 3: Additional metrics over prediction steps. Bars and stars indicate steps where PatchTST significantly outperforms Naive and AR, respectively (one-sided t-test, $p < 0.05$). PatchTST loss exceeds 95% of the Average model loss (solid line) after 1.80s (top), 1.65s (middle), 1.97s (bottom). AR loss exceeds 95% of the Average model loss after 1.28s (top), 1.23s (middle), 1.60s (bottom).
• Figure 4: Performance across PatchTST models trained with different forecast horizons. PatchTST (18), PatchTST (35), PatchTST (70): PatchTST trained with forecast horizon of 18, 35, and 70 steps.