Logo-LLM: Local and Global Modeling with Large Language Models for Time Series Forecasting

TL;DR

Logo-LLM tackles time series forecasting by exploiting hierarchical representations in a pre-trained LLM: shallow layers encode local dynamics while deeper layers capture global trends. It decouples these signals via Local-Mixer and Global-Mixer modules and aligns them with patch-based temporal inputs, using a frozen GPT-2 backbone to maintain efficiency. Across long-term, few-shot, and zero-shot benchmarks, Logo-LLM achieves superior or competitive accuracy with low additional computation, demonstrating strong data efficiency and cross-domain generalization. This work highlights the value of multi-layer LLM semantics for multi-scale temporal modeling and opens avenues for more principled, layer-aware forecasting with frozen backbones.

Abstract

Time series forecasting is critical across multiple domains, where time series data exhibits both local patterns and global dependencies. While Transformer-based methods effectively capture global dependencies, they often overlook short-term local variations in time series. Recent methods that adapt large language models (LLMs) into time series forecasting inherit this limitation by treating LLMs as black-box encoders, relying solely on the final-layer output and underutilizing hierarchical representations. To address this limitation, we propose Logo-LLM, a novel LLM-based framework that explicitly extracts and models multi-scale temporal features from different layers of a pre-trained LLM. Through empirical analysis, we show that shallow layers of LLMs capture local dynamics in time series, while deeper layers encode global trends. Moreover, Logo-LLM introduces lightweight Local-Mixer and Global-Mixer modules to align and integrate features with the temporal input across layers. Extensive experiments demonstrate that Logo-LLM achieves superior performance across diverse benchmarks, with strong generalization in few-shot and zero-shot settings while maintaining low computational overhead.

Figures (7)

• Figure 1: Comparison of LLM usage paradigms. Prior works treat LLMs as black-box encoders and use only the last-layer feature for prediction. In contrast, our method explicitly extracts features from multiple layers, leveraging shallow-layer features for local modeling and deep-layer features for global modeling, enabling a more fine-grained understanding of temporal dynamics.
• Figure 2: Overview of the proposed Logo-LLM framework. Logo-LLM extracts intermediate representations from multiple layers of a pre-trained LLM to explicitly model local and global temporal patterns. Two specialized Mixer modules are introduced to align these hierarchical features with the temporal input, enabling fine-grained modeling of local and global variations. Most LLM parameters are kept frozen, enabling efficient adaptation under limited supervision.
• Figure 3: Comparison of Logo-LLM and CALF with various numbers of layers on ETTh1 and ETTh2 datasets. The prediction length is set as {96, 192} with input length $L=96$.
• Figure 4: Visualization of different selections {1, 2, 3, 4, 5, 6} about local feature layer on ETTh1, ETTm2 and ETTh2 datasets. The prediction length is set as 96 with input length $L=96$.
• Figure 5: We visualize the similarity matrices for each patch of multiple Transformer layers in Logo-LLM, showing that shallow layers exhibit pronounced local patterns while deeper layers capture broader global dependencies. The prediction length is set as 96 with input length L = 96.