Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep TPC: Temporal-Prior Conditioning for Time Series Forecasting

Filippos Bellos, NaveenJohn Premkumar, Yannis Avrithis, Nam H. Nguyen, Jason J. Corso

TL;DR

The paper tackles long-horizon multivariate time-series forecasting with restricted parameter budgets by addressing the shallow handling of temporal information in LLM-based approaches. It introduces Temporal-Prior Conditioning (TPC), which uses small, learnable TS-tokens that repeatedly cross-attend to temporal embeddings derived from compact textual prompts processed by a frozen LLM, across multiple decoder depths. By training only the cross-attention modules, TS-tokens, and the output head, the method achieves state-of-the-art or competitive results on diverse datasets while maintaining parameter efficiency. The work demonstrates that elevating time to a first-class modality yields robust temporal reasoning and practical gains for long-term forecasting, with code available at the provided repository.

Abstract

LLM-for-time series (TS) methods typically treat time shallowly, injecting positional or prompt-based cues once at the input of a largely frozen decoder, which limits temporal reasoning as this information degrades through the layers. We introduce Temporal-Prior Conditioning (TPC), which elevates time to a first-class modality that conditions the model at multiple depths. TPC attaches a small set of learnable time series tokens to the patch stream; at selected layers these tokens cross-attend to temporal embeddings derived from compact, human-readable temporal descriptors encoded by the same frozen LLM, then feed temporal context back via self-attention. This disentangles time series signal and temporal information while maintaining a low parameter budget. We show that by training only the cross-attention modules and explicitly disentangling time series signal and temporal information, TPC consistently outperforms both full fine-tuning and shallow conditioning strategies, achieving state-of-the-art performance in long-term forecasting across diverse datasets. Code available at: https://github.com/fil-mp/Deep_tpc

Deep TPC: Temporal-Prior Conditioning for Time Series Forecasting

TL;DR

The paper tackles long-horizon multivariate time-series forecasting with restricted parameter budgets by addressing the shallow handling of temporal information in LLM-based approaches. It introduces Temporal-Prior Conditioning (TPC), which uses small, learnable TS-tokens that repeatedly cross-attend to temporal embeddings derived from compact textual prompts processed by a frozen LLM, across multiple decoder depths. By training only the cross-attention modules, TS-tokens, and the output head, the method achieves state-of-the-art or competitive results on diverse datasets while maintaining parameter efficiency. The work demonstrates that elevating time to a first-class modality yields robust temporal reasoning and practical gains for long-term forecasting, with code available at the provided repository.

Abstract

LLM-for-time series (TS) methods typically treat time shallowly, injecting positional or prompt-based cues once at the input of a largely frozen decoder, which limits temporal reasoning as this information degrades through the layers. We introduce Temporal-Prior Conditioning (TPC), which elevates time to a first-class modality that conditions the model at multiple depths. TPC attaches a small set of learnable time series tokens to the patch stream; at selected layers these tokens cross-attend to temporal embeddings derived from compact, human-readable temporal descriptors encoded by the same frozen LLM, then feed temporal context back via self-attention. This disentangles time series signal and temporal information while maintaining a low parameter budget. We show that by training only the cross-attention modules and explicitly disentangling time series signal and temporal information, TPC consistently outperforms both full fine-tuning and shallow conditioning strategies, achieving state-of-the-art performance in long-term forecasting across diverse datasets. Code available at: https://github.com/fil-mp/Deep_tpc
Paper Structure (10 sections, 3 equations, 1 figure, 2 tables)

This paper contains 10 sections, 3 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Preliminaries
  4. Input encoding
  5. Temporal embedding generation
  6. Temporal conditioning
  7. Next-token prediction
  8. Experiments
  9. Discussion
  10. Acknowledgments

Figures (1)

  • Figure 1: TPC Overview. TPC modules are inserted at L layers of a frozen LLM. TS-tokens (yellow) mediate between time series patch embeddings (blue) and temporal embeddings (green) by performing cross-attention to gather temporal information, which is then distributed to patches via self-attention.