A Dataset for Spatiotemporal-Sensitive POI Question Answering

TL;DR

POI-QA addresses the lack of spatiotemporal reasoning in POI QA by constructing a large bilingual benchmark from GAIA trajectories and Chengdu POIs. The dataset creation pipeline comprises geographic annotation, trajectory-based POI mapping with privacy-preserving anonymization, and generation of spatiotemporal QA pairs across multiple granularity levels, yielding over 5 million samples. Baseline experiments with Llama3.1 and Qwen2.5 using zero-shot, LoRA, and RAG reveal substantial gaps to human performance, especially on fine-grained subcategories and open-ended generation. POI-QA provides a robust benchmark for developing spatiotemporal-aware, multilingual QA systems and is publicly accessible on Kaggle.

Abstract

Spatiotemporal relationships are critical in data science, as many prediction and reasoning tasks require analysis across both spatial and temporal dimensions--for instance, navigating an unfamiliar city involves planning itineraries that sequence locations and timing cultural experiences. However, existing Question-Answering (QA) datasets lack sufficient spatiotemporal-sensitive questions, making them inadequate benchmarks for evaluating models' spatiotemporal reasoning capabilities. To address this gap, we introduce POI-QA, a novel spatiotemporal-sensitive QA dataset centered on Point of Interest (POI), constructed through three key steps: mining and aligning open-source vehicle trajectory data from GAIA with high-precision geographic POI data, rigorous manual validation of noisy spatiotemporal facts, and generating bilingual (Chinese/English) QA pairs that reflect human-understandable spatiotemporal reasoning tasks. Our dataset challenges models to parse complex spatiotemporal dependencies, and evaluations of state-of-the-art multilingual LLMs (e.g., Qwen2.5-7B, Llama3.1-8B) reveal stark limitations: even the top-performing model (Qwen2.5-7B fine-tuned with RAG+LoRA) achieves a top 10 Hit Ratio (HR@10) of only 0.41 on the easiest task, far below human performance at 0.56. This underscores persistent weaknesses in LLMs' ability to perform consistent spatiotemporal reasoning, while highlighting POI-QA as a robust benchmark to advance algorithms sensitive to spatiotemporal dynamics. The dataset is publicly available at https://www.kaggle.com/ds/7394666.

Figures (11)

• Figure 1: A toy example of spatiotemporal sensitive questions.
• Figure 2: QA sample synthesizing.
• Figure 3: Transformer-based Decoder Structure
• Figure 4: Training Procedure of Llama3.1
• Figure 5: The overview of the retrieval process.