Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QKCV Attention: Enhancing Time Series Forecasting with Static Categorical Embeddings for Both Lightweight and Pre-trained Foundation Models

Hao Wang, Baojun Ma

TL;DR

QKCV Attention introduces static categorical embeddings into dot-product attention to better capture category-aware patterns in time-series forecasting. By injecting C into the K stream, and providing three variants (v1–v3) alongside a static covariate encoder, the approach serves as a versatile plug-in for Transformer-based forecasts and pretrained TimeFM models. Empirical results on Meal, Favorita, and M5 show consistent accuracy gains (lower WPE, P50, P90) and substantial memory savings during finetuning, with TFT often achieving the best performance. The work advances practical deployment by enabling efficient fine-tuning of foundation models and offering interpretable insights via shifted feature importances and attention patterns.

Abstract

In real-world time series forecasting tasks, category information plays a pivotal role in capturing inherent data patterns. This paper introduces QKCV (Query-Key-Category-Value) attention, an extension of the traditional QKV framework that incorporates a static categorical embedding C to emphasize category-specific information. As a versatile plug-in module, QKCV enhances the forecasting accuracy of attention-based models (e.g., Vanilla Transformer, Informer, PatchTST, TFT) across diverse real-world datasets. Furthermore, QKCV demonstrates remarkable adaptability in fine-tuning univariate time series foundation model by solely updating the static embedding C while preserving pretrained weights, thereby reducing computational overhead and achieving superior fine-tuning performance.

QKCV Attention: Enhancing Time Series Forecasting with Static Categorical Embeddings for Both Lightweight and Pre-trained Foundation Models

TL;DR

QKCV Attention introduces static categorical embeddings into dot-product attention to better capture category-aware patterns in time-series forecasting. By injecting C into the K stream, and providing three variants (v1–v3) alongside a static covariate encoder, the approach serves as a versatile plug-in for Transformer-based forecasts and pretrained TimeFM models. Empirical results on Meal, Favorita, and M5 show consistent accuracy gains (lower WPE, P50, P90) and substantial memory savings during finetuning, with TFT often achieving the best performance. The work advances practical deployment by enabling efficient fine-tuning of foundation models and offering interpretable insights via shifted feature importances and attention patterns.

Abstract

In real-world time series forecasting tasks, category information plays a pivotal role in capturing inherent data patterns. This paper introduces QKCV (Query-Key-Category-Value) attention, an extension of the traditional QKV framework that incorporates a static categorical embedding C to emphasize category-specific information. As a versatile plug-in module, QKCV enhances the forecasting accuracy of attention-based models (e.g., Vanilla Transformer, Informer, PatchTST, TFT) across diverse real-world datasets. Furthermore, QKCV demonstrates remarkable adaptability in fine-tuning univariate time series foundation model by solely updating the static embedding C while preserving pretrained weights, thereby reducing computational overhead and achieving superior fine-tuning performance.
Paper Structure (21 sections, 6 equations, 7 figures, 4 tables)

This paper contains 21 sections, 6 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Model intuition
  5. Formulation
  6. Implementations
  7. Static feature embedding
  8. Attention variants
  9. Experiments
  10. Real-world datasets
  11. Training
  12. Results and performance
  13. Chart analysis
  14. Conclusions
  15. Technical Appendices and Supplementary Material
  16. ...and 6 more sections

Figures (7)

  • Figure 1: A comparison between the Dot Product Attention mechanism (left) and our proposed modified QKCV attention mechanism incorporating static categorical embeddings (middle). The figure on the right shows a typical structure of a Transformer variant and the location of the attention module (in red) that we modified.
  • Figure 2: Our modified TFT structure (a) and TimeFM (b) serves as examples of applying the QKCV mechanism to attention-based lightweight model and pre-trained fundation model, where static embedding vector C, derived from Static Covariates Encoders, is directly fed into the Multi-head Attention layer, as indicated by the black dashed line. The code of both models and others are available in the GitHub repository.
  • Figure 3: Averaged real values versus forecast outputs from TimeFM for each dataset.
  • Figure 4: Feature importances for static imbeddings in TFT.
  • Figure 5: Analysis of QKCV attention mechanisms within TimeFM on M5 dataset. (a-c) Learned static embeddings (C) across three QKCV variants, from top 200 samples; (d-f) Associated attention score (QK) matrices of top 100 samples for corresponding subsets. All samples are consistently sorted by entity ID across subfigures. Complete results for all datasets are available in Supplementary Materials.
  • ...and 2 more figures