Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

STD-PLM: Understanding Both Spatial and Temporal Properties of Spatial-Temporal Data with PLM

YiHeng Huang, Xiaowei Mao, Shengnan Guo, Yubin Chen, Junfeng Shen, Tiankuo Li, Youfang Lin, Huaiyu Wan

TL;DR

STD-PLM tackles the dual challenge of spatial-temporal forecasting and imputation in a zero-shot and few-shot setting by adapting a pre-trained language model to spatial-temporal data. It introduces spatial-temporal embeddings, separate spatial and temporal tokenizers, and a sandglass attention mechanism to model high-order correlations while maintaining efficiency; a constrained loss aligns region-level tokens with graph structure during fine-tuning with LoRA. Empirical results on four PEMS datasets show competitive forecasting and state-of-the-art imputation performance, with notable few-shot and zero-shot capabilities, highlighting practical deployment potential with limited data. The work suggests that a unified PLM-based approach can enable rapid, scalable spatial-temporal reasoning for real-world applications like traffic management and planning, while also outlining limitations and directions for scaling to larger PLMs and broader datasets.

Abstract

Spatial-temporal forecasting and imputation are important for real-world intelligent systems. Most existing methods are tailored for individual forecasting or imputation tasks but are not designed for both. Additionally, they are less effective for zero-shot and few-shot learning. While pre-trained language model (PLM) have exhibited strong pattern recognition and reasoning abilities across various tasks, including few-shot and zero-shot learning, their applications in spatial-temporal data understanding has been constrained by insufficient modeling of complex correlations such as the temporal correlations, spatial connectivity, non-pairwise and high-order spatial-temporal correlations within data. In this paper, we propose STD-PLM for understanding both spatial and temporal properties of \underline{S}patial-\underline{T}emporal \underline{D}ata with \underline{PLM}, which is capable of implementing both spatial-temporal forecasting and imputation tasks. STD-PLM understands spatial-temporal correlations via explicitly designed spatial and temporal tokenizers. Topology-aware node embeddings are designed for PLM to comprehend and exploit the topology structure of data in inductive manner. Furthermore, to mitigate the efficiency issues introduced by the PLM, we design a sandglass attention module (SGA) combined with a specific constrained loss function, which significantly improves the model's efficiency while ensuring performance. Extensive experiments demonstrate that STD-PLM exhibits competitive performance and generalization capabilities across the forecasting and imputation tasks on various datasets. Moreover, STD-PLM achieves promising results on both few-shot and zero-shot tasks. The code is made available at \href{https://github.com/Hyheng/STD-PLM}{https://github.com/Hyheng/STD-PLM}

STD-PLM: Understanding Both Spatial and Temporal Properties of Spatial-Temporal Data with PLM

TL;DR

STD-PLM tackles the dual challenge of spatial-temporal forecasting and imputation in a zero-shot and few-shot setting by adapting a pre-trained language model to spatial-temporal data. It introduces spatial-temporal embeddings, separate spatial and temporal tokenizers, and a sandglass attention mechanism to model high-order correlations while maintaining efficiency; a constrained loss aligns region-level tokens with graph structure during fine-tuning with LoRA. Empirical results on four PEMS datasets show competitive forecasting and state-of-the-art imputation performance, with notable few-shot and zero-shot capabilities, highlighting practical deployment potential with limited data. The work suggests that a unified PLM-based approach can enable rapid, scalable spatial-temporal reasoning for real-world applications like traffic management and planning, while also outlining limitations and directions for scaling to larger PLMs and broader datasets.

Abstract

Spatial-temporal forecasting and imputation are important for real-world intelligent systems. Most existing methods are tailored for individual forecasting or imputation tasks but are not designed for both. Additionally, they are less effective for zero-shot and few-shot learning. While pre-trained language model (PLM) have exhibited strong pattern recognition and reasoning abilities across various tasks, including few-shot and zero-shot learning, their applications in spatial-temporal data understanding has been constrained by insufficient modeling of complex correlations such as the temporal correlations, spatial connectivity, non-pairwise and high-order spatial-temporal correlations within data. In this paper, we propose STD-PLM for understanding both spatial and temporal properties of \underline{S}patial-\underline{T}emporal \underline{D}ata with \underline{PLM}, which is capable of implementing both spatial-temporal forecasting and imputation tasks. STD-PLM understands spatial-temporal correlations via explicitly designed spatial and temporal tokenizers. Topology-aware node embeddings are designed for PLM to comprehend and exploit the topology structure of data in inductive manner. Furthermore, to mitigate the efficiency issues introduced by the PLM, we design a sandglass attention module (SGA) combined with a specific constrained loss function, which significantly improves the model's efficiency while ensuring performance. Extensive experiments demonstrate that STD-PLM exhibits competitive performance and generalization capabilities across the forecasting and imputation tasks on various datasets. Moreover, STD-PLM achieves promising results on both few-shot and zero-shot tasks. The code is made available at \href{https://github.com/Hyheng/STD-PLM}{https://github.com/Hyheng/STD-PLM}
Paper Structure (26 sections, 9 equations, 1 figure, 11 tables)

This paper contains 26 sections, 9 equations, 1 figure, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Definition
  5. Model Structure
  6. Spatial-Temporal Embedding.
  7. Spatial-Temporal Tokenizer.
  8. Sandglass Attention.
  9. Unified Output Projection.
  10. Model Training.
  11. Experiments
  12. Datasets
  13. Experimental Settings
  14. Overall Performance
  15. Few-shot and Zero-shot Performance
  16. ...and 11 more sections

Figures (1)

  • Figure 1: Model architecture. Module A provides spatial-temporal embeddings for node and temporal information. The spatial-temporal tokenizers B construct temporal and spatial tokens from different perspectives. Module C builds region-level spatial tokens based on the node-level spatial tokens which is the output of B-2. Module D projects the hidden representations to target output. For the PLM, We utilize partial freezing and LoRA to fine-tune the multi-head attention, position embedding and layernorm layers.