Enhancing Large Language Models for Time-Series Forecasting via Vector-Injected In-Context Learning

TL;DR

This work tackles the challenge of using large language models for time-series forecasting (TSF) without costly fine-tuning. It introduces LVICL, a vector-injected in-context learning framework that freezes all LLM parameters and derives a context vector from time-series exemplars using a learnable adapter; this context is refined and injected into the residual stream across all LLM layers during forecasting. LVICL achieves strong forecasting performance across diverse real-world datasets, offering competitive results with substantially lower GPU memory than full fine-tuning and outperforming many partial-fine-tuning baselines. The method also demonstrates reduced sensitivity to example selection/order and good zero-shot transfer, marking LVICL as a practical, scalable approach for TSF with large language models.

Abstract

The World Wide Web needs reliable predictive capabilities to respond to changes in user behavior and usage patterns. Time series forecasting (TSF) is a key means to achieve this goal. In recent years, the large language models (LLMs) for TSF (LLM4TSF) have achieved good performance. However, there is a significant difference between pretraining corpora and time series data, making it hard to guarantee forecasting quality when directly applying LLMs to TSF; fine-tuning LLMs can mitigate this issue, but often incurs substantial computational overhead. Thus, LLM4TSF faces a dual challenge of prediction performance and compute overhead. To address this, we aim to explore a method for improving the forecasting performance of LLM4TSF while freezing all LLM parameters to reduce computational overhead. Inspired by in-context learning (ICL), we propose LVICL. LVICL uses our vector-injected ICL to inject example information into a frozen LLM, eliciting its in-context learning ability and thereby enhancing its performance on the example-related task (i.e., TSF). Specifically, we first use the LLM together with a learnable context vector adapter to extract a context vector from multiple examples adaptively. This vector contains compressed, example-related information. Subsequently, during the forward pass, we inject this vector into every layer of the LLM to improve forecasting performance. Compared with conventional ICL that adds examples into the prompt, our vector-injected ICL does not increase prompt length; moreover, adaptively deriving a context vector from examples suppresses components harmful to forecasting, thereby improving model performance. Extensive experiments demonstrate the effectiveness of our approach.

Figures (10)

• Figure 1: A schematic diagram illustrating the computation process of the $l$-th layer of a Transformer from the perspective of the residual stream.
• Figure 2: Comparison of forecasting performance in three settings: 1) full-parameter fine-tuning; 2) freezing the LLM and fine-tuning only the input and output layers (w/o ICL); 3) introducing ICL on top of setting 2 (with ICL).
• Figure 3: The overall framework of the proposed LVICL. (a) The acquisition and injection of context vectors. (b) The LLM-based time series forecasting model.
• Figure 4: Impact of post-processing strategies for context vectors.
• Figure 5: Impact of the injection location of the context vector.