$\textbf{S}^2$IP-LLM: Semantic Space Informed Prompt Learning with LLM for Time Series Forecasting

TL;DR

S^2IP-LLM introduces a semantic-space informed prompting framework for time series forecasting that aligns pre-trained LLM semantic space with time-series embeddings through a decomposition-based tokenization and top-K semantic anchor prompts. By deriving semantic anchors from word embeddings and matching them to time-series embeddings via cosine similarity, the method retrieves informative prefix prompts that guide a frozen or lightly tuned LLM backbone (GPT-2) to forecast across diverse datasets and horizons. The approach is validated on long-term, short-term, and few-shot tasks, with ablations confirming the value of semantic-space guidance, patch-based tokenization, and the joint space alignment. The findings suggest a practical pathway to leverage foundation models for robust, cross-domain time series forecasting with limited task-specific fine-tuning, offering improvements over state-of-the-art baselines and strong transferability across data regimes.

Abstract

Recently, there has been a growing interest in leveraging pre-trained large language models (LLMs) for various time series applications. However, the semantic space of LLMs, established through the pre-training, is still underexplored and may help yield more distinctive and informative representations to facilitate time series forecasting. To this end, we propose Semantic Space Informed Prompt learning with LLM ($S^2$IP-LLM) to align the pre-trained semantic space with time series embeddings space and perform time series forecasting based on learned prompts from the joint space. We first design a tokenization module tailored for cross-modality alignment, which explicitly concatenates patches of decomposed time series components to create embeddings that effectively encode the temporal dynamics. Next, we leverage the pre-trained word token embeddings to derive semantic anchors and align selected anchors with time series embeddings by maximizing the cosine similarity in the joint space. This way, $S^2$IP-LLM can retrieve relevant semantic anchors as prompts to provide strong indicators (context) for time series that exhibit different temporal dynamics. With thorough empirical studies on multiple benchmark datasets, we demonstrate that the proposed $S^2$IP-LLM can achieve superior forecasting performance over state-of-the-art baselines. Furthermore, our ablation studies and visualizations verify the necessity of prompt learning informed by semantic space.

Figures (7)

• Figure 1: The demonstration of semantic space informed prompting in $S^{2}$IP-LLM. The input time series is decomposed and mapped to obtain time series (TS) embedding. Next, the TS embedding is aligned with semantic anchors derived from the pre-trained word token embedding. Finally, top-$K$ similar semantic anchors are retrieved and used as prefix-prompts with TS embedding.
• Figure 2: The model architecture of $S^{2}\text{IP-LLM}$. The input time series is normalized, decomposed, patched individually, and concatenated to represent the context of time series (TS). Semantic space informed prompting performs alignment between the contextual TS embeddings and the semantic anchors extracted from pre-trained word embeddings, and retrieves the most similar $K$ ones as prefix-prompts. The decomposed TS representations from pre-trained LLM are linearly projected and combined as the TS forecast.
• Figure 3: Parameter sensitivity analysis in predicting 96 and 192 steps: (1) and (4) show the effect of prompt length on ETTh2 and ETTm2 datasets; (2) and (5) show the effect of alignment coefficient $\lambda$ on ETTh2 and ETTm2 datasets; (3) and (6) show the effect of semantic space size $V^{\prime}$ on ETTh2 and ETTm2 datasets.
• Figure 4: The t-SNE and PCA plots of embeddings space: blue: semantic anchor embeddings; red: time series embeddings; orange: prefix-prompted time series embeddings
• Figure 5: The t-SNE and PCA plots prefix-prompted time series embeddings with different $\lambda$