Rethinking Time Series Forecasting with LLMs via Nearest Neighbor Contrastive Learning

TL;DR

This paper introduces NNCL-TLLM, a method to repurpose large language models for time-series forecasting by learning time-series compatible text prototypes and formulating prompts through nearest-neighbor contrastive learning. The approach aligns word-token embeddings with time-series characteristics and finetunes only the layer normalization and positional embeddings to preserve the LLM prior while reducing trainable parameters. Through a patch-based time-series representation and a shared objective that combines forecasting loss with prototype and contrastive losses, NNCL-TLLM delivers strong few-shot performance and competitive long- and short-term forecasting across diverse datasets. The results demonstrate the practical potential of LLM-based time-series forecasting with principled prompt learning, though future work is needed to capture channel dependencies in multivariate settings and to further explore efficiency gains.

Abstract

Adapting Large Language Models (LLMs) that are extensively trained on abundant text data, and customizing the input prompt to enable time series forecasting has received considerable attention. While recent work has shown great potential for adapting the learned prior of LLMs, the formulation of the prompt to finetune LLMs remains challenging as prompt should be aligned with time series data. Additionally, current approaches do not effectively leverage word token embeddings which embody the rich representation space learned by LLMs. This emphasizes the need for a robust approach to formulate the prompt which utilizes the word token embeddings while effectively representing the characteristics of the time series. To address these challenges, we propose NNCL-TLLM: Nearest Neighbor Contrastive Learning for Time series forecasting via LLMs. First, we generate time series compatible text prototypes such that each text prototype represents both word token embeddings in its neighborhood and time series characteristics via end-to-end finetuning. Next, we draw inspiration from Nearest Neighbor Contrastive Learning to formulate the prompt while obtaining the top-$k$ nearest neighbor time series compatible text prototypes. We then fine-tune the layer normalization and positional embeddings of the LLM, keeping the other layers intact, reducing the trainable parameters and decreasing the computational cost. Our comprehensive experiments demonstrate that NNCL-TLLM outperforms in few-shot forecasting while achieving competitive or superior performance over the state-of-the-art methods in long-term and short-term forecasting tasks.

Figures (4)

• Figure 1: Overview of the proposed neighborhood aware time series compatible text prototype (TCTP) learning. The TCTPs are learned such that, each TCTP is a representative of the word token embeddings in its neighborhood. The learning process minimizes the distance between each word token embedding and its nearest TCTP embedding. The TCTP embedding $e1$ moves to reduce the distance between its neighborhood word token embeddings: $w1$ and $w2$. The text prototypes become time series compatible via end-to-end finetuning of the framework.
• Figure 2: Illustration of NNCL-TLLM architecture. Given the univariate time series input we obtain the time series embedding $Z^{(i)}$ using a patch embedding layer and a linear layer. LLM word token embeddings are used to generate neighborhood aware time series compatible text prototypes (TCTPs). We use NNCL by inserting the TCTPs into a support set at the end of each training step. The top-$k$ nearest neighbor TCTPs from the support set is obtained and we compute the contrastive loss; ($L_{\text{NNCL}}$) between the time series embedding and the top-$k$ nearest neighbor TCTPs. Finally, we formulate the prompt to finetune the LLM by concatenating the time series patch embeddings and top-$k$ nearest neighbor TCTPs from the support set.
• Figure 3: Illustration of the UMAP visualizations of neighborhood aware TCTP embeddings in 2-dimensional space before (a) and after (b) optimization.
• Figure 4: Illustration of the analysis of parameter sensitivity. MSE values are reported for ETTh1 dataset for [96, 192] prediction horizons.