Semantic-Enhanced Time-Series Forecasting via Large Language Models

TL;DR

SE-LLM addresses the modality gap between time-series data and linguistic representations by introducing Temporal-Semantic Cross-Correlation (TSCC) and Time-Adapter to fuse temporal structure into LLM-based forecasting while keeping the backbone LLM frozen for efficiency. The TSCC component uses cross-attention and AM-VAE-based anomaly modeling to enrich the semantic space with temporal patterns, and the Time-Adapter injects long- and short-term temporal dependencies into attention via parallel LSTMs. A sliding-window preprocessing further reduces computational costs, enabling scalable forecasting across diverse datasets and horizons. Empirical results across ETTh1, Weather, Traffic, ECL, Solar, and M4 demonstrate state-of-the-art accuracy, robust zero-shot generalization, and notable efficiency gains over prior methods.

Abstract

Time series forecasting plays a significant role in finance, energy, meteorology, and IoT applications. Recent studies have leveraged the generalization capabilities of large language models (LLMs) to adapt to time series forecasting, achieving promising performance. However, existing studies focus on token-level modal alignment, instead of bridging the intrinsic modality gap between linguistic knowledge structures and time series data patterns, greatly limiting the semantic representation. To address this issue, we propose a novel Semantic-Enhanced LLM (SE-LLM) that explores the inherent periodicity and anomalous characteristics of time series to embed into the semantic space to enhance the token embedding. This process enhances the interpretability of tokens for LLMs, thereby activating the potential of LLMs for temporal sequence analysis. Moreover, existing Transformer-based LLMs excel at capturing long-range dependencies but are weak at modeling short-term anomalies in time-series data. Hence, we propose a plugin module embedded within self-attention that models long-term and short-term dependencies to effectively adapt LLMs to time-series analysis. Our approach freezes the LLM and reduces the sequence dimensionality of tokens, greatly reducing computational consumption. Experiments demonstrate the superiority performance of our SE-LLM against the state-of-the-art (SOTA) methods.

Figures (6)

• Figure 1: (a) The input time series is mapped to obtain time series (TS) Embeddings. Next, the feature space is aligned with the semantic space derived from the pre-trained word token embedding. (b) Temporal patterns are injected into the joint space and use implicit prompts to LLM for forecasting. (c) Embedding Time Adapter enhances learning of temporal patterns.
• Figure 2: (a) An overview of SE-LLM. Red indicates trainable, while blue indicates frozen. (b) The overall framework of TSCC, with detailed processes referring to Fig. \ref{['12']}. (c) For the Transformer-based LLM, the Time-Adapter is embedded within the key and value vectors of the multi-head attention mechanism. JS is an abbreviation for Joint Space.
• Figure 3: Architecture of TSCC module, where $\mathrm{\overline{DA}}$ and $\mathrm{\overline{DC}}$ correspond to "de-anomaly semantic" and "anomaly semantic".
• Figure 4: The architecture of Time-Adapter, a modular enhancement designed to improve the temporal modeling capabilities of LLMs. It comprises two linear layers and two parallel LSTM modules, enabling effective capture of both long-term and short-term temporal dependencies.
• Figure 5: Qualitative analysis illustrating how the AM-VAE effectively models anomaly patterns.