Instructing the Architecture Search for Spatial-temporal Sequence Forecasting with LLM

TL;DR

The paper addresses STSF by automating neural architecture search with an LLM-guided framework, ISTS, that leverages background knowledge and problem structure to improve search efficiency. It introduces a multi-level enhancement: step-level prompting to leverage LLM internal knowledge, instance-level quick evaluation with a memory bank, and a task-level two-stage search to avoid local optima. The search space comprises four-node DAG architectures built from three cell types (STP, STT, TTS) using a high-order MixGC spatial operator and POLLA temporal operator, with a memory-augmented generation mechanism and COT/TOT prompting. Experiments on five real-world datasets show ISTS achieves competitive or superior accuracy compared with NAS and non-NAS baselines, while offering substantial efficiency gains (e.g., inference-only LLM guiding search reduces time by about 92% in some setups). Ablation studies confirm the necessity of memory accumulation and the two-stage exploration for robust performance, and analyses show stability across different open-source LLMs, highlighting ISTS’s practical potential for efficient STSF NAS.

Abstract

Spatial-temporal sequence forecasting (STSF) is a long-standing research problem with widespread real-world applications. Neural architecture search (NAS), which automates the neural network design, has been shown effective in tackling the STSF problem. However, the existing NAS methods for STSF focus on generating architectures in a time-consuming data-driven fashion, which heavily limits their ability to use background knowledge and explore the complicated search trajectory. Large language models (LLMs) have shown remarkable ability in decision-making with comprehensive internal world knowledge, but how it could benefit NAS for STSF remains unexplored. In this paper, we propose a novel NAS method for STSF based on LLM. Instead of directly generate architectures with LLM, We inspire the LLM's capability with a multi-level enhancement mechanism. Specifically, on the step-level, we decompose the generation task into decision steps with powerful prompt engineering and inspire LLM to serve as instructor for architecture search based on its internal knowledge. On the instance-level, we utilize a one-step tuning framework to quickly evaluate the architecture instance and a memory bank to cumulate knowledge to improve LLM's search ability. On the task-level, we propose a two-stage architecture search, balancing the exploration stage and optimization stage, to reduce the possibility of being trapped in local optima. Extensive experimental results demonstrate that our method can achieve competitive effectiveness with superior efficiency against existing NAS methods for STSF.

Figures (11)

• Figure 1: Directly use LLM for NAS V.S. Our multi-level LLM-based NAS. Existing methods simply use the LLM as an end-to-end architecture generator. However, our proposed method involves LLM deeply by multiple-level enhancement to improve and inspire LLM's decision-making ability. (a) Step-level: Decompose the task into logically coherent steps with powerful prompting mechanism to lead LLM to instruct the generation and selection of the candidate architecture based on its internal world knowledge, allowing LLM to quickly find well-performed architectures. (b) Instance-level: A one-step evaluation framework simulates the architecture candidates quickly and accumulate experience with a memory bank to inspire LLM to explore better architectures. (c) Task-level: Two-stage search mechanism assists the large model in determining the optimal results and reduce the possibility of being trapped into local optima.
• Figure 2: Cells and the architecture for searching. (a) 3 types of cells with different combination of spatial and temporal operators provide possibility to capture various spatial-temporal mixed dependencies. (b) Architecture combines multiple cells in a DAG way, and each cell has 3 kinds of option as indicated in (a).
• Figure 3: The framework of our proposed ISTS method. (a) LLM instructed architecture generation: Powerful task decomposing prompt encourage LLM to generate new architecture candidates sensibly its policy with LLM's internal knowledge and comprehension and quickly find well-performed architectures. (b) Experience gain and cumulate framework: Once a valid architecture candidate generated, the framework construct the architecture and quickly tune and evaluate it with several training data. The sorted memory bank store the architecture evaluation results to enhance LLM by providing experience accumulation. (c) Two-stage search mechanism: To escape from local optimal results, we encourage LLM to explore new architectures though may not be well-performed ones in the first stage. Then, with a wide exploration, the only target for the second stage is to improve performance.
• Figure 4: The performance comparison on the PEMS-04 dataset indicating that our method consistently achieves high performance in MAE and RMSE and is competitive in MAPE.
• Figure 5: Evaluation of the exploration rates.