Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distilling Time Series Foundation Models for Efficient Forecasting

Yuqi Li, Kuiye Ding, Chuanguang Yang, Szu-Yu Chen, Yingli Tian

TL;DR

DistilTS addresses the high cost of Time-Series Foundation Models by introducing horizon-weighted distillation and a factorized temporal alignment mechanism. The horizon-weighted objective rebalance supervision to improve long-horizon fidelity, while the alignment module bridges the gap between variate-wise student embeddings and point-wise teacher representations. Empirical results on five real-world datasets show DistilTS matches or closely approaches full-scale TSFMs with orders of magnitude fewer parameters and dramatically faster inference. The work provides an open-source framework for applying TSFM-specific distillation to enable practical, scalable forecasting.

Abstract

Time Series foundation models (TSFMs) deliver strong forecasting performance through large-scale pretraining, but their large parameter sizes make deployment costly. While knowledge distillation offers a natural and effective approach for model compression, techniques developed for general machine learning tasks are not directly applicable to time series forecasting due to the unique characteristics. To address this, we present DistilTS, the first distillation framework specifically designed for TSFMs. DistilTS addresses two key challenges: (1) task difficulty discrepancy, specific to forecasting, where uniform weighting makes optimization dominated by easier short-term horizons, while long-term horizons receive weaker supervision; and (2) architecture discrepancy, a general challenge in distillation, for which we design an alignment mechanism in the time series forecasting. To overcome these issues, DistilTS introduces horizon-weighted objectives to balance learning across horizons, and a temporal alignment strategy that reduces architectural mismatch, enabling compact models. Experiments on multiple benchmarks demonstrate that DistilTS achieves forecasting performance comparable to full-sized TSFMs, while reducing parameters by up to 1/150 and accelerating inference by up to 6000x. Code is available at: https://github.com/itsnotacie/DistilTS-ICASSP2026.

Distilling Time Series Foundation Models for Efficient Forecasting

TL;DR

DistilTS addresses the high cost of Time-Series Foundation Models by introducing horizon-weighted distillation and a factorized temporal alignment mechanism. The horizon-weighted objective rebalance supervision to improve long-horizon fidelity, while the alignment module bridges the gap between variate-wise student embeddings and point-wise teacher representations. Empirical results on five real-world datasets show DistilTS matches or closely approaches full-scale TSFMs with orders of magnitude fewer parameters and dramatically faster inference. The work provides an open-source framework for applying TSFM-specific distillation to enable practical, scalable forecasting.

Abstract

Time Series foundation models (TSFMs) deliver strong forecasting performance through large-scale pretraining, but their large parameter sizes make deployment costly. While knowledge distillation offers a natural and effective approach for model compression, techniques developed for general machine learning tasks are not directly applicable to time series forecasting due to the unique characteristics. To address this, we present DistilTS, the first distillation framework specifically designed for TSFMs. DistilTS addresses two key challenges: (1) task difficulty discrepancy, specific to forecasting, where uniform weighting makes optimization dominated by easier short-term horizons, while long-term horizons receive weaker supervision; and (2) architecture discrepancy, a general challenge in distillation, for which we design an alignment mechanism in the time series forecasting. To overcome these issues, DistilTS introduces horizon-weighted objectives to balance learning across horizons, and a temporal alignment strategy that reduces architectural mismatch, enabling compact models. Experiments on multiple benchmarks demonstrate that DistilTS achieves forecasting performance comparable to full-sized TSFMs, while reducing parameters by up to 1/150 and accelerating inference by up to 6000x. Code is available at: https://github.com/itsnotacie/DistilTS-ICASSP2026.
Paper Structure (12 sections, 9 equations, 3 figures, 3 tables)

This paper contains 12 sections, 9 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. DistilTS Framework
  4. Horizon-weighted knowledge distillation
  5. Factorized Temporal Alignment
  6. Boundary Case Analysis
  7. Alternative KD Variants.
  8. EXPERIMENT
  9. Experimental Settings
  10. Main Results
  11. Ablation Experiments
  12. Conclusion

Figures (3)

  • Figure 1: Overview of DistilTS. Horizon-weighted objectives rebalance supervision across horizons, and a factorized temporal alignment module projects variate-wise student embeddings into the teacher’s point-wise space.
  • Figure 2: Experiment: Efficiency vs. Performance. The bubble plot presents each model variant separately , while DistilTS is shown as $DistilTS_{L}$ (DLinear dlinear as student) and $DistilTS_{T}$ (iTransformer liu2023itransformer as student). The accompanying table reports the averaged parameter counts and forecasting errors (MSE) within each model family, with DistilTS values averaged over its two variants.
  • Figure 3: Ablation studies of DistilTS under different teacher settings. Left: Comparison across different TSFMs as teachers, showing that the choice of teacher architecture significantly impacts distillation performance. Right: Comparison within the same architecture (TimeMoe) at different scales (50M vs. 200M), showing that increasing teacher size provides only limited additional benefits.