SVTime: Small Time Series Forecasting Models Informed by "Physics" of Large Vision Model Forecasters

TL;DR

SVTime introduces a physics-inspired approach to long-term time-series forecasting by distilling inductive biases from large vision-model forecasters into compact linear architectures. By encoding inter-period consistency, patch-wise variety, and distance-attenuating local attention, and coupling them with a backcast-residual decomposition, SVTime(-t) achieves large-model-like accuracy with orders of magnitude fewer parameters. Extensive experiments across 8 datasets and 21 baselines demonstrate strong performance even without pre-training, while ablations and comparisons to knowledge-distillation highlight the efficiency and practicality for resource-constrained environments. The work offers a practical path to sustainable, high-performance forecasting that leverages task-aligned biases rather than raw scale.

Abstract

Time series AI is crucial for analyzing dynamic web content, driving a surge of pre-trained large models known for their strong knowledge encoding and transfer capabilities across diverse tasks. However, given their energy-intensive training, inference, and hardware demands, using large models as a one-fits-all solution raises serious concerns about carbon footprint and sustainability. For a specific task, a compact yet specialized, high-performing model may be more practical and affordable, especially for resource-constrained users such as small businesses. This motivates the question: Can we build cost-effective lightweight models with large-model-like performance on core tasks such as forecasting? This paper addresses this question by introducing SVTime, a novel Small model inspired by large Vision model (LVM) forecasters for long-term Time series forecasting (LTSF). Recently, LVMs have been shown as powerful tools for LTSF. We identify a set of key inductive biases of LVM forecasters -- analogous to the "physics" governing their behaviors in LTSF -- and design small models that encode these biases through meticulously crafted linear layers and constraint functions. Across 21 baselines spanning lightweight, complex, and pre-trained large models on 8 benchmark datasets, SVTime outperforms state-of-the-art (SOTA) lightweight models and rivals large models with 10^3 fewer parameters than LVMs, while enabling efficient training and inference in low-resource settings.

Figures (8)

• Figure 1: An overview of (a) forecasting performance vs. inference time on ETTm1 dataset, where circle size reflects model size, which is a zoom-in of the small models in (b); in (b), TimeLLMjin2024time is ignored for its much larger size and longer inference time (see $\S$\ref{['sec.exp.large_models']}); and (c) is a categorization of small, complex, and pre-trained large models.
• Figure 2: An illustration of inter-period consistency.
• Figure 3: An illustration of patch-wise variety.
• Figure 4: An illustration of (a) distance-attenuating local attention; and (b) our annealing constraint function.
• Figure 5: An overview of SVTime framework. (a) SVTime(-t) uses a backcast-residual decomposition to adaptively learn trend and seasonal components. (b) LVM-IB block uses linear layers and a constraint function to encode IB1-IB3 ($\S$\ref{['sec.ib1']}-$\S$\ref{['sec.ib3']}).