Understanding the Implicit Biases of Design Choices for Time Series Foundation Models

TL;DR

The paper tackles how design knobs in time series foundation models—specifically patch size, embedding strategy, and training objective—induce implicit biases that influence forecasting beyond benchmark performance. Using theory and controlled experiments, it identifies three core biases: temporal (frequency and periodicity learning from patching), geometric (embedding-induced locality and scale effects), and regression-to-the-mean (loss-induced centralization vs. sharp distributions). It demonstrates these biases via comparisons (e.g., Chronos vs Chronos-Bolt) and a case study on outlier handling, showing complex interactions and regime-dependent benefits. The findings offer practical guidance for selecting design choices to achieve forward-compatibility with more data and compute, and contribute to understanding simple model principles at scale in TSFMs.

Abstract

Time series foundation models (TSFMs) are a class of potentially powerful, general-purpose tools for time series forecasting and related temporal tasks, but their behavior is strongly shaped by subtle inductive biases in their design. Rather than developing a new model and claiming that it is better than existing TSFMs, e.g., by winning on existing well-established benchmarks, our objective is to understand how the various ``knobs'' of the training process affect model quality. Using a mix of theory and controlled empirical evaluation, we identify several design choices (patch size, embedding choice, training objective, etc.) and show how they lead to implicit biases in fundamental model properties (temporal behavior, geometric structure, how aggressively or not the model regresses to the mean, etc.); and we show how these biases can be intuitive or very counterintuitive, depending on properties of the model and data. We also illustrate in a case study on outlier handling how multiple biases can interact in complex ways; and we discuss implications of our results for learning the bitter lesson and building TSFMs.

Figures (33)

• Figure 1: An illustration of the three broad classes of biases that we identify and investigate. The temporal bias tends to be induced by patching, which transplants the temporal dependencies from the input sequence to a Transformer's hidden space; the geometric bias tends to be related to the input embedding, where a discrete versus a continuous embedding preserves the input space's geometry differently; the regression-to-the-mean bias tends to be caused by the training loss, where a continuous loss tends to average different outcomes more than a cross-entropy loss does. We also illustrate the interaction of these biases in a case study of outlier handling.
• Figure 2: In (a), we show the forecasts of a signal consisting of a low-frequency and a high-frequency mode using two pretrained Chronos-Bolt models with patch size $k = 1$ and $k = 16$, respectively. In (b), we vary the frequencies of the two modes, and we evaluate the frequency loss in predicting each mode. We see that Chronos-Bolt with a patch size of $16$ fails significantly at capturing the high-frequency information. In (c), we project the embeddings of low-frequency patches and high-frequency patches onto the two principal components, and we show how the attention matrices align with these two principal components on average. We observe that the attention scores are heavily dominated by the low-frequency patches.
• Figure 3: In (a), we show the autocorrelation in the output of a Transformer given a periodic input context, which measures the periodicity preservation of a model. In (b), we show leading principal components of the encoded context of a Chronos-Bolt model, with or without the [REG] token, given a periodic input. In (c) , we show the best matching score, aggregated on $10,000$ randomly sampled time series in Chronos' training dataset, with or without patching. The best matching score is computed by comparing the last motif of length $64$ to all previous motifs.
• Figure 4: Panel (a) shows the angles between the embedded vectors $\boldsymbol{\phi}(x)$ and $\boldsymbol{\phi}(y)$ for Chronos' embedding $\boldsymbol{\phi}_Q$ and Chronos-Bolt's embedding $\boldsymbol{\phi}_C$, respectively. Panel (b) shows a histogram of all self-attention scores when inferring the two models on a corpus of contexts. Panel (c) shows the forecasts from both models on a context formed by repeating a motif of a chaotic system. Chronos achieves a significantly lower MASE than Chronos-Bolt. The cross-attention scores are aggregated across all layers when generating the first prediction token. See \ref{['app:geomtric']} for details.
• Figure 5: Panel (a) shows the distance between the embedded vectors for Chronos' embedding $\boldsymbol{\phi}_Q$ and Chronos-Bolt's embedding $\boldsymbol{\phi}_C$. Panel (b) shows Chronos' and Chronos-Bolt's forecasts of a multi-scale time series. Panel (c) quantifies the forecasting quality by computing the relative MSE, as we increase the ratio between the two scales. See \ref{['app:geomtric']} for more details.

Theorems & Definitions (2)

• Theorem 1
• proof : Proof of \ref{['thm.frequencybias']}