SEMPO: Lightweight Foundation Models for Time Series Forecasting

TL;DR

SEMPO introduces a lightweight time series foundation model that nests two innovations—Energy-aware Spectral Decomposition (EASD) and Mixture-of-Prompts enabled Transformer (MoPFormer)—to maximize data efficiency and transferability. Pretraining on a modest 83M time points with 6.5M parameters, SEMPO achieves strong zero-shot and few-shot generalization across 16 datasets, outperforming much larger models while reducing computational burden. The approach demonstrates that explicit modeling of low-energy spectral components and token-level prompt routing can capture heterogeneous temporal patterns without bloating model size. This work offers a practical path toward deployable, generalizable time series FMs suitable for resource-constrained environments, with implications for broader cross-domain forecasting scenarios.

Abstract

The recent boom of large pre-trained models witnesses remarkable success in developing foundation models (FMs) for time series forecasting. Despite impressive performance across diverse downstream forecasting tasks, existing time series FMs possess massive network architectures and require substantial pre-training on large-scale datasets, which significantly hinders their deployment in resource-constrained environments. In response to this growing tension between versatility and affordability, we propose SEMPO, a novel lightweight foundation model that requires pretraining on relatively small-scale data, yet exhibits strong general time series forecasting. Concretely, SEMPO comprises two key modules: 1) energy-aware SpEctral decomposition module, that substantially improves the utilization of pre-training data by modeling not only the high-energy frequency signals but also the low-energy yet informative frequency signals that are ignored in current methods; and 2) Mixture-of-PrOmpts enabled Transformer, that learns heterogeneous temporal patterns through small dataset-specific prompts and adaptively routes time series tokens to prompt-based experts for parameter-efficient model adaptation across different datasets and domains. Equipped with these modules, SEMPO significantly reduces both pre-training data scale and model size, while achieving strong generalization. Extensive experiments on two large-scale benchmarks covering 16 datasets demonstrate the superior performance of SEMPO in both zero-shot and few-shot forecasting scenarios compared with state-of-the-art methods. Code and data are available at https://github.com/mala-lab/SEMPO.

Figures (12)

• Figure 1: Left: Most time series FMs cannot effectively utilize low-energy frequency signals, exemplified by Chronos$_S$DBLP:journals/corr/abs-2403-07815 on the ETTh1 dataset DBLP:conf/aaai/ZhouZPZLXZ21. Middle: The EASD module in SEMPO helps mitigate this issue. Right: The number of parameters in SEMPO and state-of-the-art (SOTA) time series FMs against average zero-/few-shot performance on the TSLib benchmark DBLP:conf/iclr/WuHLZ0L23.
• Figure 2: SEMPO follows an encoder-decoder architecture. Given a univariate time series, ① its EASD module transforms it into the frequency domain and partitions into high- and low-energy components. ② These components are independently masked across multiple frequencies, fused, and transformed back to the time domain. After patching and projection, the masked representation is fed to backbone layers and a reconstruction head as part of the energy-aware pre-training. During MoP tuning, all previous modules are frozen; only the MoP and the prediction head are trainable. The masked input is then ③ routed to token-dependent prompts via an adaptive router and ④ concatenated into the key and value of self-attention, enabling prompt-guided reconstruction and prediction.
• Figure 3: Zero-shot results on the GIFT-Eval benchmark. Full details in Appendix \ref{['appendix_D']}.
• Figure 4: Spectrum visualizations of predictions from Chronos$_S$, Moirai$_L$, and SEMPO on Electricity.
• Figure 5: Gating scores of various experts.