Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reprogramming Foundational Large Language Models(LLMs) for Enterprise Adoption for Spatio-Temporal Forecasting Applications: Unveiling a New Era in Copilot-Guided Cross-Modal Time Series Representation Learning

Sakhinana Sagar Srinivas, Chidaksh Ravuru, Geethan Sannidhi, Venkataramana Runkana

TL;DR

The paper tackles the challenge of accurate MTSF in enterprise settings by reprogramming foundational LLMs for time-series tasks. It introduces LLM-TS Net, a cross-modal framework that combines on-premise, fine-tuned open-source LLMs with traditional time-series encoders, dynamic prompting, and a Grouped-query Multi-head Attention (GQ-MHA) backbone to model intra- and inter-series dependencies. Key innovations include LoRA-AMR for memory-efficient fine-tuning on consumer hardware, a dynamic prompt pool with retrieval-based adaptation, and cross-modal integration of textual trend descriptions with time-series embeddings, enabling uncertainty estimation via a Gaussian likelihood (w/Unc-LLM-TS Net). Empirical results on PeMS and METR datasets show significant improvements over baselines in MAE, RMSE, and MAPE across multiple horizons, with robust performance under missing data and clear evidence that each component (LLM processing, dynamic prompting, intra-/inter-series modeling, and CMA) contributes meaningfully to predictive accuracy. The work demonstrates a practical, privacy-preserving path to enterprise-grade spatio-temporal forecasting through a memory-efficient, cross-domain, transformer-based architecture that leverages both textual descriptions and numerical patterns for robust forecasting and uncertainty quantification.

Abstract

Spatio-temporal forecasting plays a crucial role in various sectors such as transportation systems, logistics, and supply chain management. However, existing methods are limited by their ability to handle large, complex datasets. To overcome this limitation, we introduce a hybrid approach that combines the strengths of open-source large and small-scale language models (LLMs and LMs) with traditional forecasting methods. We augment traditional methods with dynamic prompting and a grouped-query, multi-head attention mechanism to more effectively capture both intra-series and inter-series dependencies in evolving nonlinear time series data. In addition, we facilitate on-premises customization by fine-tuning smaller open-source LMs for time series trend analysis utilizing descriptions generated by open-source large LMs on consumer-grade hardware using Low-Rank Adaptation with Activation Memory Reduction (LoRA-AMR) technique to reduce computational overhead and activation storage memory demands while preserving inference latency. We combine language model processing for time series trend analysis with traditional time series representation learning method for cross-modal integration, achieving robust and accurate forecasts. The framework effectiveness is demonstrated through extensive experiments on various real-world datasets, outperforming existing methods by significant margins in terms of forecast accuracy.

Reprogramming Foundational Large Language Models(LLMs) for Enterprise Adoption for Spatio-Temporal Forecasting Applications: Unveiling a New Era in Copilot-Guided Cross-Modal Time Series Representation Learning

TL;DR

The paper tackles the challenge of accurate MTSF in enterprise settings by reprogramming foundational LLMs for time-series tasks. It introduces LLM-TS Net, a cross-modal framework that combines on-premise, fine-tuned open-source LLMs with traditional time-series encoders, dynamic prompting, and a Grouped-query Multi-head Attention (GQ-MHA) backbone to model intra- and inter-series dependencies. Key innovations include LoRA-AMR for memory-efficient fine-tuning on consumer hardware, a dynamic prompt pool with retrieval-based adaptation, and cross-modal integration of textual trend descriptions with time-series embeddings, enabling uncertainty estimation via a Gaussian likelihood (w/Unc-LLM-TS Net). Empirical results on PeMS and METR datasets show significant improvements over baselines in MAE, RMSE, and MAPE across multiple horizons, with robust performance under missing data and clear evidence that each component (LLM processing, dynamic prompting, intra-/inter-series modeling, and CMA) contributes meaningfully to predictive accuracy. The work demonstrates a practical, privacy-preserving path to enterprise-grade spatio-temporal forecasting through a memory-efficient, cross-domain, transformer-based architecture that leverages both textual descriptions and numerical patterns for robust forecasting and uncertainty quantification.

Abstract

Spatio-temporal forecasting plays a crucial role in various sectors such as transportation systems, logistics, and supply chain management. However, existing methods are limited by their ability to handle large, complex datasets. To overcome this limitation, we introduce a hybrid approach that combines the strengths of open-source large and small-scale language models (LLMs and LMs) with traditional forecasting methods. We augment traditional methods with dynamic prompting and a grouped-query, multi-head attention mechanism to more effectively capture both intra-series and inter-series dependencies in evolving nonlinear time series data. In addition, we facilitate on-premises customization by fine-tuning smaller open-source LMs for time series trend analysis utilizing descriptions generated by open-source large LMs on consumer-grade hardware using Low-Rank Adaptation with Activation Memory Reduction (LoRA-AMR) technique to reduce computational overhead and activation storage memory demands while preserving inference latency. We combine language model processing for time series trend analysis with traditional time series representation learning method for cross-modal integration, achieving robust and accurate forecasts. The framework effectiveness is demonstrated through extensive experiments on various real-world datasets, outperforming existing methods by significant margins in terms of forecast accuracy.
Paper Structure (20 sections, 21 equations, 5 figures, 7 tables)

This paper contains 20 sections, 21 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Problem Definition
  3. LoRA-AMR:
  4. Fine-Tuning small-scale LMs:
  5. Dynamic Prompting Mechanism Design:
  6. Modeling Intra-series dependencies:
  7. Modeling Inter-series dependencies:
  8. Output Layer:
  9. DATASETS
  10. EXPERIMENTAL RESULTS
  11. CONCLUSION
  12. Technical Appendix
  13. Additional Datasets and Experimental Results
  14. Ablation Study
  15. Forecasting Uncertainity
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Our framework incorporates the joint optimization of three methods: (a) a sequential stack of dynamic prompting mechanism to transfer relevant historical knowledge for adapting to new trends, coupled with learning intra- and inter-series dependencies to obtain contextualized time-series embeddings; (b) the utilization of large-scale model descriptions on time series trends to fine-tune a smaller language model, which then generates text-level embeddings encapsulating these trends; and (c) an output layer modeled with the multi-head attention (MHA) mechanism for integrating cross-domain embeddings and facilitating time series forecasting. This joint optimization framework provides a comprehensive and robust approach to modeling and forecasting spatio-temporal MTS data, enhancing adaptability, accuracy, and efficiency, and is designed for better generalization and scalability in real-world forecasting tasks.
  • Figure 2: The illustration of (a) full-model fine-tuning (FT), (b) LoRA, and (c) LoRA-AMR.
  • Figure 3: The figure shows the comparative performance of different forecasting models on the METR-LA dataset. It showcases the accuracy and precision of each model in predicting traffic flow trends, emphasizing their respective strengths and limitations.
  • Figure 4: The figure illustrates the performance comparison of various models on the PEMS-BAY dataset. It highlights the forecast accuracy and error margins for each model, providing insights into their relative effectiveness in predicting traffic flow patterns.
  • Figure 5: The table shows the pointwise prediction error for multi-horizon forecasting tasks on benchmark datasets.