Benchmarking Spatiotemporal Reasoning in LLMs and Reasoning Models: Capabilities and Challenges

TL;DR

STARK presents a hierarchical, multi-task benchmark for spatiotemporal reasoning in CPS, spanning state estimation, state-relationship reasoning, and world-knowledge-enabled tasks across 26 scenarios with 14,552 challenges. It rigorously compares 8 LLMs and 3 LRMs using direct answers and Python Code Interpreter tool usage, revealing that LRMs generally outperform LLMs on localization and tracking, while LLMs can close the gap on knowledge-heavy tasks. The framework integrates formal spatial and temporal logics ($DE-9IM$ and $Allen's interval algebra$) and emphasizes tool usage as a core competency for CPS agents. Larger reasoning models (notably the o3 family) achieve leading performance, and CI-based reasoning offers both gains and risks, motivating guardrails and targeted fine-tuning for robust spatiotemporal CPS performance. By exposing clear limitations and strengths, STARK guideposts architectural and methodological innovations for intelligent CPS systems.

Abstract

Spatiotemporal reasoning plays a key role in Cyber-Physical Systems (CPS). Despite advances in Large Language Models (LLMs) and Large Reasoning Models (LRMs), their capacity to reason about complex spatiotemporal signals remains underexplored. This paper proposes a hierarchical SpatioTemporal reAsoning benchmaRK, STARK, to systematically evaluate LLMs across three levels of reasoning complexity: state estimation (e.g., predicting field variables, localizing and tracking events in space and time), spatiotemporal reasoning over states (e.g., inferring spatial-temporal relationships), and world-knowledge-aware reasoning that integrates contextual and domain knowledge (e.g., intent prediction, landmark-aware navigation). We curate 26 distinct spatiotemporal tasks with diverse sensor modalities, comprising 14,552 challenges where models answer directly or by Python Code Interpreter. Evaluating 3 LRMs and 8 LLMs, we find LLMs achieve limited success in tasks requiring geometric reasoning (e.g., multilateration or triangulation), particularly as complexity increases. Surprisingly, LRMs show robust performance across tasks with various levels of difficulty, often competing or surpassing traditional first-principle-based methods. Our results show that in reasoning tasks requiring world knowledge, the performance gap between LLMs and LRMs narrows, with some LLMs even surpassing LRMs. However, the LRM o3 model continues to achieve leading performance across all evaluated tasks, a result attributed primarily to the larger size of the reasoning models. STARK motivates future innovations in model architectures and reasoning paradigms for intelligent CPS by providing a structured framework to identify limitations in the spatiotemporal reasoning of LLMs and LRMs.

Figures (11)

• Figure 1: Three-Tiered Architecture of STARK.
• Figure 2: State estimation. Four types of sensor modalities for localization and tracking.
• Figure 3: Illustration of Tier 2 and Tier 3 challenges
• Figure 4: Model ranks. Further from the center indicates better performance.
• Figure 5: Benchmark results aggregated over task. Model performance is measured by RMSPE (field variable prediction) and RMSE (others). The numbers in each box plot represent the median.