STRIDE-QA: Visual Question Answering Dataset for Spatiotemporal Reasoning in Urban Driving Scenes

TL;DR

This work tackles the gap in spatiotemporal reasoning for autonomous driving by introducing STRIDE-QA, a large-scale ego-centric VQA dataset with dense 3D groundings derived from multi-sensor driving data. It defines three QA tasks that require both spatial grounding and short-horizon prediction and demonstrates that fine-tuning Vision-Language Models on STRIDE-QA yields substantial improvements over baselines, including near-present-mense localization (LSR) and improved temporal consistency (MLSR, TLC). The dataset comprises 285K frames and 16M QA pairs across 100+ hours of Tokyo driving, with automated annotations from 3D detection, tracking, segmentation, and visibility filtering. Overall, STRIDE-QA provides a rigorous benchmark to push physically grounded, ego-centric spatiotemporal reasoning in autonomous systems and guides future work toward robust planning and safety-critical decision-making.

Abstract

Vision-Language Models (VLMs) have been applied to autonomous driving to support decision-making in complex real-world scenarios. However, their training on static, web-sourced image-text pairs fundamentally limits the precise spatiotemporal reasoning required to understand and predict dynamic traffic scenes. We address this critical gap with STRIDE-QA, a large-scale visual question answering (VQA) dataset for physically grounded reasoning from an ego-centric perspective. Constructed from 100 hours of multi-sensor driving data in Tokyo, capturing diverse and challenging conditions, STRIDE-QA is the largest VQA dataset for spatiotemporal reasoning in urban driving, offering 16 million QA pairs over 285K frames. Grounded by dense, automatically generated annotations including 3D bounding boxes, segmentation masks, and multi-object tracks, the dataset uniquely supports both object-centric and ego-centric reasoning through three novel QA tasks that require spatial localization and temporal prediction. Our benchmarks demonstrate that existing VLMs struggle significantly, achieving near-zero scores on prediction consistency. In contrast, VLMs fine-tuned on STRIDE-QA exhibit dramatic performance gains, achieving 55% success in spatial localization and 28% consistency in future motion prediction, compared to near-zero scores from general-purpose VLMs. Therefore, STRIDE-QA establishes a comprehensive foundation for developing more reliable VLMs for safety-critical autonomous systems.

Figures (9)

• Figure 1: STRIDE-QA is a large-scale VQA dataset for spatiotemporal reasoning in autonomous driving, comprising 285 K frames and 16 M QA pairs from over 100 hours of urban driving in Tokyo. (a) It includes multi-view RGB images and (b) LiDAR point clouds, processed via a modular pipeline with 3D object detection, segmentation, tracking, and visibility filtering to produce spatially and temporally grounded annotations. (c) The annotations enable object-centric, ego-centric spatial, and spatiotemporal QA tasks, allowing structured evaluation of physically grounded reasoning over time.
• Figure 2: Example data from our STRIDE-QA dataset. From top to bottom, each QA pair corresponds to (A) Object-centric Spatial QA, (B) Ego-centric Spatial QA, and (C) Ego-centric Spatiotemporal QA.
• Figure 3: Qualitative results on Spatiotemporal QA Benchmark. Across four driving scenarios and predicted future frames at $t = 0\text{--}3\,s$, our fine-tuned STRIDE-Qwen2.5-VL-7B consistently delivers more accurate distance and angle estimates than GPT-4o, highlighting its superior spatiotemporal reasoning.
• Figure 4: Comparison of localization performance on a polar coordinate system (angle=heading direction, radius=distance from ego). Top row (Qwen2.5-VL-7B-Instruct): The baseline’s sparse, biased predictions indicate a simplistic, memorized behavior. Bottom row (STRIDE-Qwen2.5-VL-7B): Our fine-tuned model generates dense, plausible forecasts, achieving nearly perfect localizations at $t =0\,s$. Green dots denote success (LSR), red denotes failure, and the blue wedge marks the camera's $\pm30^{\circ}$ FOV.
• Figure 5: LSR trends across dynamic scenarios, grouped by out-of-view (OOV) likelihood. The sharp performance drop in scenarios with a high OOV likelihood (blue background) compared to in-view scenarios (grey background) highlights that OOV prediction is the primary challenge.