Towards Neural Scaling Laws for Time Series Foundation Models

TL;DR

This work investigates neural scaling laws for Time Series Foundation Models (TSFMs) under both ID and OOD conditions, comparing encoder-only and decoder-only Transformer architectures across model size, compute, and data. The authors demonstrate that negative log-likelihood loss scales approximately as a power law with these factors on both ID and OOD data, while encoder-only models tend to be more scalable and advanced TSFMs mainly boost ID performance at the potential cost of OOD scalability. By deploying a large, curated pre-training corpus and evaluating on robust ID/OOD benchmarks, they reveal that data scaling yields stronger OOD improvements in MAPE, whereas model size is the primary driver of performance. The work culminates in design principles and guidance for scaling TSFMs, highlighting that larger models with appropriate architectural choices offer practical paths to improved generalization across unseen domains. Key results are formalized with power-law relationships and empirical comparisons among Moirai, Chronos, and baseline encoders/decoders, providing actionable direction for allocating compute, data, and architectural investments in TSFM development.

Abstract

Scaling laws offer valuable insights into the design of time series foundation models (TSFMs). However, previous research has largely focused on the scaling laws of TSFMs for in-distribution (ID) data, leaving their out-of-distribution (OOD) scaling behavior and the influence of model architectures less explored. In this work, we examine two common TSFM architectures, encoder-only and decoder-only Transformers, and investigate their scaling behavior on both ID and OOD data. These models are trained and evaluated across varying parameter counts, compute budgets, and dataset sizes. Our experiments reveal that the log-likelihood loss of TSFMs exhibits similar scaling behavior in both OOD and ID settings. We further compare the scaling properties across different architectures, incorporating two state-of-the-art TSFMs as case studies, showing that model architecture plays a significant role in scaling. The encoder-only Transformers demonstrate better scalability than the decoder-only Transformers, while the architectural enhancements in the two advanced TSFMs primarily improve ID performance but reduce OOD scalability. While scaling up TSFMs is expected to drive performance breakthroughs, the lack of a comprehensive understanding of TSFM scaling laws has hindered the development of a robust framework to guide model scaling. We fill this gap in this work by synthesizing our findings and providing practical guidelines for designing and scaling larger TSFMs with enhanced model capabilities.

Figures (19)

• Figure 1: Architectures of Baseline Time Series Foundation Models. As the most widely used two Transformer architectures, encoder-only Transformer and decoder-only Transformer are selected to as our baseline. A time series is divided into multiple patches, each treated as a token and fed into the Transformer model. The shaded patches represent the future horizon to be predicted.
• Figure 2: Parameter Scaling. The scaling effect of total trainable model parameters on the in-distribution (ID) and out-of-distribution (OOD) forecasting performance, which is evaluated using NLL and MAPE metrics. When evaluated with NLL, both ID and OOD results follow an approximate power law scaling with parameter count, exhibiting consistent trends across different data distributions. The blue and red horizontal dashed lines represent the baselines of the exponential smoothing (ETS) forecasting method.
• Figure 3: Compute Scaling. The computation scaling results indicate that model performance scales approximately according to a power law with increasing compute, consistent across both ID and OOD scenarios. The ID and OOD results illustrate that there is an lower bound for loss and MAPE on both test data under a given computational budget.
• Figure 4: Data Scaling. The blue and red plots illustrate how data volume affects the ID and OOD forecasting performance of encoder-only Transformers, evaluated using NLL and MAPE metrics. The results indicate that in both scenarios, model performance scales approximately as a power law with data volume.
• Figure 5: Scaling Laws of Encoder-only vs. Decoder-only Transformer. This figure presents a comparison of scaling behaviors on NLL between encoder-only and decoder-only Transformer across three different axes: number of parameters, compute, and dataset size. Overall, both models exhibit similar scalability patterns with respect to model parameters, computation and dataset sizes across ID and OOD data, but differ in ID performance.