Beyond Accuracy: Are Time Series Foundation Models Well-Calibrated?

TL;DR

This paper addresses the calibration of time series foundation models (TSFMs) by evaluating five leading TSFMs and two baselines on six univariate datasets. It analyzes how prediction heads, autoregressive strategies, and horizon length affect probabilistic calibration using metrics such as $\text{PCE}$, $\text{CCE}$, and $\text{SIW}$, revealing that TSFMs are generally better calibrated than baselines and are not systematically over- or under-confident. A Gaussian prediction head is notably under-confident, while quantile, Student's $t$, and mixture heads show similar, robust calibration; long-term forecasting benefits from longer horizons and trajectory-based autoregression. The findings inform practical deployment of TSFMs, emphasizing calibration-aware design choices for reliable uncertainty quantification in real-world forecasting, and point to future work on multivariate settings, distribution shifts, and higher quantiles. The study thus advances understanding of uncertainty quantification in TSFMs and their readiness for decision-critical applications, beyond mere point accuracy.

Abstract

The recent development of foundation models for time series data has generated considerable interest in using such models across a variety of applications. Although foundation models achieve state-of-the-art predictive performance, their calibration properties remain relatively underexplored, despite the fact that calibration can be critical for many practical applications. In this paper, we investigate the calibration-related properties of five recent time series foundation models and two competitive baselines. We perform a series of systematic evaluations assessing model calibration (i.e., over- or under-confidence), effects of varying prediction heads, and calibration under long-term autoregressive forecasting. We find that time series foundation models are consistently better calibrated than baseline models and tend not to be either systematically over- or under-confident, in contrast to the overconfidence often seen in other deep learning models.

Figures (16)

• Figure 1: Weighted Quantile Loss (WQL) calibration metric can be highly correlated with point accuracy error (MASE). Results shown for WQL (left) and PCE (right) plotted against MASE on the Glucose dataset used in our experiments. Background markers are results for each model and each individual time series; larger centroids are medians across time series. On this dataset, WQL incorrectly identifies ARIMA as the best calibrated model.
• Figure 2: TSFMs offer competitive point forecasting performance compared to the baseline models, often having better accuracy than the baselines.$y$-axis: Mean Absolute Scaled Error (MASE) measuring point accuracy error of the median prediction (lower is better), Glucose and Patents use their own $y$-axis scale (on the right). The error bars are computed as the Standard Error of the Mean (SEM) across timeseries.
• Figure 3: TSFMs are better calibrated than the baseline models and are not systematically over- or under-confident. Top: Probabilistic Calibration Error (PCE) across datasets and models using the default quantile projection block. (Patents PCE uses its own $y$-axis scale (on the right)). Lower PCE values are better. Bottom: Centered Calibration Error (CCE) evaluating systematic overconfidence (positive) or under-confidence (negative).
• Figure 4: Gaussian prediction heads are under-confident and are in most cases outperformed by the other prediction heads. Calibration properties of various prediction heads with Centered Calibration Error (CCE) on the $x$-axis and Probabilistic Calibration Error (PCE) on the $y$-axis. Large positive CCE values indicate overconfident predictions, while large negative CCE values show under-confidence. Background markers are the results for each head (shape) with each backbone model (color), while the larger gray centroids are the mean results across models.
• Figure 5: For long-term AR forecasting, increasing the horizon length and using the trajectory AR method produces better calibrated forecasts. The figure compares Probabilistic Calibration Error (PCE) for long-term forecasting using autoregression on the $y$-axis with AR prediction horizon on the $x$-axis. The color of the line depicts the model used and the line style indicates the AR method. The pink and blue dashed horizontal lines are the PCE for TiRex and Yinglong without using AR.