Transfer Learning with Foundational Models for Time Series Forecasting using Low-Rank Adaptations

TL;DR

The paper tackles how Foundation Models and Large Language Models can be repurposed for Time Series Forecasting through transfer learning. It introduces LLIAM, a LoRA-based, decoder-only LLaMA-1 7B model guided by a TSF-oriented textual prompting scheme, with a task update $\Delta W = \frac{\alpha}{r} BA$ and training limited to $A,B$. Through a two-stage evaluation—comparing against DL baselines and TimeLLM, then conducting a zero-shot generalization study on unseen domains—it demonstrates competitive SMAPE and robust generalization, while highlighting efficiency gains and reduced training overhead conducive to Green AI. The results suggest that simple, targeted LLM adaptations can leverage cross-domain temporal patterns, reducing the need for extensive feature engineering and architecture search. The work points to future exploration of larger models and broader PEFT strategies to further improve TSF performance with sustainable computing practices.

Abstract

Foundational Models are an emerging widely used technique of GenAI. These models are distinguished by their scalability and the ease with which they can be adapted through the exploitation of Transfer Learning. The availability of high computational power and large datasets have supported their development, achieving a high generalization capacity due to the enormous and heterogeneous amounts of data used in their initial training. These characteristics contribute to a solid base that can be adapted or adjusted to a wide range of tasks, increasing their applicability. This study proposes the methodology LLIAM, a straightforward adaptation of a kind of FM, Large Language Models, for the Time Series Forecasting task. An adequate time-series prompting schema and Low-Rank Adaptations are used to enhance the knowledge of the model with diverse time series datasets, known as the fine-tuning phase. A study divided in two stages has been performed for evaluating the effectiveness of the proposed methodology. Initially, a comparison was made between the performance of LLIAM and different state-of-the-art DL algorithms, including Recurrent Neural Networks and Temporal Convolutional Networks, as well as a LLM-based method, TimeLLM. Following this, a zero-shot study is presented in order to evaluate the generalization capacity of the proposed methodology with time series datasets from unknown domains not considered in the model training. The outcomes of this investigation demonstrate the efficacy of LLIAM, highlighting that this straightforward and general approach can attain competent results without the necessity for applying complex modifications. This work also encourages the use of available resources (such as these pre-trained models) and efficient fine-tuning techniques to avoid unnecessary and costly training, narrowing the gap between the goals of traditional AI and Green AI.

Figures (3)

• Figure 1: Detailed baseline Transformer architecture. Encoders capture the relationships between input tokens using multi-head attention and enhance their representations with feed-forward layers. Decoders relate the output of the encoder stack to the predicted tokens using masked and cross-attention mechanisms. Both use positional encoding. Post-processing phase (red block) is usually used to predict the next token according to the task using the output of the decoder stack. Figure adapted from Vaswani2017attention
• Figure 2: LLIAM architecture. Only Low-Rank adaptations applied to Query (Q) and Values (V) are trained. The rest of the models is freezed. Our prompting scheme is applied to the numerical input data to align with the textual representation used by the LLM.
• Figure 3: Critical differences diagram with Wilcoxon post-hoc analysis using the SMAPE metric at $\alpha = 0.05$. Models connected by a horizontal line indicate no statistically significant difference detected by the Wilcoxon test between them. A rank for each model is also indicated next to it.