UDVideoQA: A Traffic Video Question Answering Dataset for Multi-Object Spatio-Temporal Reasoning in Urban Dynamics

TL;DR

The UDVideoQA suite, including the dataset, annotation tools, and benchmarks for both VideoQA and VideoQGen, provides a foundation for advancing robust, privacy-aware, and real-world multimodal reasoning.

Abstract

Understanding the complex, multi-agent dynamics of urban traffic remains a fundamental challenge for video language models. This paper introduces Urban Dynamics VideoQA, a benchmark dataset that captures the unscripted real-world behavior of dynamic urban scenes. UDVideoQA is curated from 16 hours of traffic footage recorded at multiple city intersections under diverse traffic, weather, and lighting conditions. It employs an event-driven dynamic blur technique to ensure privacy preservation without compromising scene fidelity. Using a unified annotation pipeline, the dataset contains 28K question-answer pairs generated across 8 hours of densely annotated video, averaging one question per second. Its taxonomy follows a hierarchical reasoning level, spanning basic understanding and attribution to event reasoning, reverse reasoning, and counterfactual inference, enabling systematic evaluation of both visual grounding and causal reasoning. Comprehensive experiments benchmark 10 SOTA VideoLMs on UDVideoQA and 8 models on a complementary video question generation benchmark. Results reveal a persistent perception-reasoning gap, showing models that excel in abstract inference often fail with fundamental visual grounding. While models like Gemini Pro achieve the highest zero-shot accuracy, fine-tuning the smaller Qwen2.5-VL 7B model on UDVideoQA bridges this gap, achieving performance comparable to proprietary systems. In VideoQGen, Gemini 2.5 Pro, and Qwen3 Max generate the most relevant and complex questions, though all models exhibit limited linguistic diversity, underscoring the need for human-centric evaluation. The UDVideoQA suite, including the dataset, annotation tools, and benchmarks for both VideoQA and VideoQGen, provides a foundation for advancing robust, privacy-aware, and real-world multimodal reasoning. UDVideoQA is available at https://ud-videoqa.github.io/UD-VideoQA/UD-VideoQA/.

Figures (25)

• Figure 1: The UDVideoQA benchmark challenges a VideoQA model to analyze a busy intersection using question-answer pairs across diverse urban traffic conditions. The model's capabilities are demonstrated across five reasoning categories: attribution, basic understanding, event reasoning, reverse reasoning, and counterfactual inference.
• Figure 2: Illustrates the pipeline for creating the UDVideoQA dataset. The process begins with traffic video recording, which is segmented and temporally clipped into $10$s clips. These clips undergo dynamic anonymity blurring. The QA taxonomy and generation module then uses model based on VideoQGen benchmark and prompt engineering to create a question pool, which undergoes rigorous human in the loop quality control and refinement.
• Figure 3: UDVideoQA dataset statistics. (a) Word frequency distribution by question type, (b) Distribution of question categories across semantic domains, including pedestrians, vehicles, and environmental signage. (c) Plot illustrating the spread of question sets across six contextual dimensions: pedestrian, temporal, vehicle, event, and signage, and (d) Frequency of questions containing key interrogative terms ("what", "where", and "is") across reasoning categories.
• Figure 4: Cross-model performance comparison on the UDVideoQA benchmark. Each bar shows the range from worst to best performance of VideoLMs used in this study.
• Figure 5: Deployment sites of traffic cameras used for data collection. The selected intersections capture diverse lighting and weather conditions, providing representative coverage of real-world urban traffic environments.