LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters

TL;DR

LLM4TS introduces a data-efficient framework that repurposes pre-trained LLMs for long-term multivariate time-series forecasting. It adopts a two-stage fine-tuning pipeline—time-series alignment with autoregressive objectives and forecasting fine-tuning—together with a novel two-level multi-scale temporal encoding to inject time-specific context into patched time-series inputs. Through extensive experiments on seven real-world datasets, LLM4TS achieves state-of-the-art or competitive performance in both full- and few-shot settings while using only a small fraction of trainable parameters via PEFT (LoRA and LayerNorm Tuning). The approach demonstrates strong representation learning capabilities and significant practical impact for scenarios with limited labeled data, with potential extensions to other tasks and LLM families in future work.

Abstract

Multivariate time-series forecasting is vital in various domains, e.g., economic planning and weather prediction. Deep train-from-scratch models have exhibited effective performance yet require large amounts of data, which limits real-world applicability. Recently, researchers have leveraged the representation learning transferability of pre-trained Large Language Models (LLMs) to handle limited non-linguistic datasets effectively. However, incorporating LLMs with time-series data presents challenges of limited adaptation due to different compositions between time-series and linguistic data, and the inability to process multi-scale temporal information. To tackle these challenges, we propose LLM4TS, a framework for time-series forecasting with pre-trained LLMs. LLM4TS consists of a two-stage fine-tuning strategy: the time-series alignment stage to align LLMs with the nuances of time-series data, and the forecasting fine-tuning stage for downstream time-series forecasting tasks. Furthermore, our framework features a novel two-level aggregation method that integrates multi-scale temporal data within pre-trained LLMs, enhancing their ability to interpret time-specific information. In experiments across 7 time-series forecasting datasets, LLM4TS is superior to existing state-of-the-art methods compared with trained-from-scratch models in full-shot scenarios, and also achieves the highest rank in few-shot scenarios. In addition, evaluations compared with different unsupervised representation learning approaches highlight LLM4TS's effectiveness with representation learning in forecasting tasks. Ablation studies further validate each component's contribution to LLM4TS and underscore the essential role of utilizing LLM's pre-trained weights for optimal performance. The code is available at https://github.com/blacksnail789521/LLM4TS.

Figures (7)

• Figure 1: Model performance comparison on few-shot forecasting.
• Figure 2: Problem formulation for multivariate time-series forecasting.
• Figure 3: LLM4TS framework. The numbers in the patched time-series (e.g., 1, 2, ..., 16 in the first patch) indicate the sequential order of the timestamps. The framework consists of two stages: (a) Time-series alignment, which uses the autoregressive approach to align the pre-trained LLM with patched time-series data. (b) Forecasting fine-tuning, which starts with linear probing (i.e., only the output layer is unfrozen), followed by full fine-tuning (all the layers and PEFT components in the LLM are unfrozen).
• Figure 4: Multi-scale temporal encoding for patched time-series data. This process involves a two-level aggregation. Here, only the first patch is shown for simplicity; in practice, all patches in a batch are processed simultaneously. Level 1 aggregation calculates the temporal embedding for each time unit and sums them together. Next, Level 2 aggregation applies a pooling method to extract the final temporal embedding.
• Figure 5: Ablation study on key components in LLM4TS. Each ablation is conducted under both full- and few-shot learning with $10\%$ training data. We report results averaged over prediction lengths $T_{out} \in \{96, 192, 336, 720\}$ for the ETTh1 dataset. The best results are in bold.