Adapting Large Language Models for Time Series Modeling via a Novel Parameter-efficient Adaptation Method

TL;DR

The paper targets the challenge of leveraging large language models for time series forecasting by introducing Time-LlaMA, a framework that tokenizes multivariate time series per channel, aligns these tokens with text prompts via cross-modal attention, and adapts the LLM backbone through a dynamic, mixture-of-experts LoRA (D-LoRA). This approach enables few-shot and zero-shot adaptation while keeping parameter additions under 1% of the backbone and maintaining efficient inference. Empirical results across diverse real-world datasets show Time-LlaMA achieving state-of-the-art performance compared to strong baselines, including recent LLM-based methods, with comprehensive ablations validating the contribution of modality alignment and the D-LoRA mechanism. The work suggests a practical, scalable path for deploying LLMs in time-series domains, offering improved accuracy and efficiency for industrial forecasting tasks.

Abstract

Time series modeling holds significant importance in many real-world applications and has been extensively studied. While pre-trained foundation models have made impressive strides in the fields of natural language processing (NLP) and computer vision (CV), their development in time series domains has been constrained by data sparsity. A series of recent studies have demonstrated that large language models (LLMs) possess robust pattern recognition and reasoning abilities over complex sequences of tokens. However, the current literature have yet striked a high-quality balance between (a) effectively aligning the time series and natural language modalities, and (b) keeping the inference efficiency. To address the above issues, we now propose the Time-LlaMA framework. Time-LlaMA first converts the time series input into token embeddings through a linear tokenization mechanism. Second, the time series token embeddings are aligned with the text prompts. Third, to further adapt the LLM backbone for time series modeling, we have developed a dynamic low-rank adaptation technique (D-LoRA). D-LoRA dynamically chooses the most suitable LoRA modules at each layer of the Transformer backbone for each time series input, enhancing the model's predictive capabilities. Our experimental results on an extensive collection of challenging real-world time series tasks confirm that our proposed method achieves the state-of-the-art (SOTA) performance.

Figures (4)

• Figure 1: Schematic illustration of our Time-LlaMA framework.
• Figure 2: Schematic illustration of our D-LoRA method.
• Figure 3: Performances under different numbers of selected LoRAs per Transformer block.
• Figure 4: Distribution of activated LoRA experts.