Efficient Multivariate Time Series Forecasting via Calibrated Language Models with Privileged Knowledge Distillation

TL;DR

This work tackles the challenge of efficient multivariate time series forecasting by introducing TimeKD, a framework that integrates calibrated language models with privileged knowledge distillation. A cross-modality teacher uses ground-truth prompts and a subtractive cross attention to extract robust future representations, while a lightweight student learns via correlation and feature distillation to enable fast inference. Extensive experiments on eight real-world datasets demonstrate TimeKD’s superior forecasting accuracy and notable efficiency gains, including strong performance in few-shot and zero-shot settings. The approach highlights the potential of combining LLM-based representations with privileged information to bridge textual and numerical data for practical MTSF applications.

Abstract

Multivariate time series forecasting (MTSF) endeavors to predict future observations given historical data, playing a crucial role in time series data management systems. With advancements in large language models (LLMs), recent studies employ textual prompt tuning to infuse the knowledge of LLMs into MTSF. However, the deployment of LLMs often suffers from low efficiency during the inference phase. To address this problem, we introduce TimeKD, an efficient MTSF framework that leverages the calibrated language models and privileged knowledge distillation. TimeKD aims to generate high-quality future representations from the proposed cross-modality teacher model and cultivate an effective student model. The cross-modality teacher model adopts calibrated language models (CLMs) with ground truth prompts, motivated by the paradigm of Learning Under Privileged Information (LUPI). In addition, we design a subtractive cross attention (SCA) mechanism to refine these representations. To cultivate an effective student model, we propose an innovative privileged knowledge distillation (PKD) mechanism including correlation and feature distillation. PKD enables the student to replicate the teacher's behavior while minimizing their output discrepancy. Extensive experiments on real data offer insight into the effectiveness, efficiency, and scalability of the proposed TimeKD.

Figures (26)

• Figure 1: Comparison of traditional teacher models and our privileged teacher. Future data is available only during training and not during testing, thus regarded as privileged information.
• Figure 2: Examples of input prompts
• Figure 3: TimeKD Framework. Cross-Modality Teacher Model processes textual prompts to reconstruct time series during the training stage. Student Model learned from the teacher model via privileged knowledge distillation for efficient forecasting.
• Figure 4: An example of calibrated attention score in the CLMs.
• Figure 5: Subtractive cross attention.