RiskCueBench: Benchmarking Anticipatory Reasoning from Early Risk Cues in Video-Language Models

TL;DR

RiskCueBench introduces a benchmark and a reproducible data curation pipeline for evaluating anticipatory reasoning in video-language models from early risk cues. By focusing on risk signal clips and employing a QRA framework, it reveals substantial gaps between state-of-the-art VLMs and human performance, especially in socially nuanced protest contexts. The study defines four evaluation metrics—risk prediction F1, reasoning grounding accuracy, temporal reasoning difference, and self-correction degradation—to dissect forecasting, grounding, and reasoning dynamics. Key findings include a reliance on static cues rather than genuine temporal reasoning, pervasive grounding failures, and degradation from excessive deliberation, highlighting critical challenges for deploying video-based risk prediction in practice.

Abstract

With the rapid growth of video centered social media, the ability to anticipate risky events from visual data is a promising direction for ensuring public safety and preventing real world accidents. Prior work has extensively studied supervised video risk assessment across domains such as driving, protests, and natural disasters. However, many existing datasets provide models with access to the full video sequence, including the accident itself, which substantially reduces the difficulty of the task. To better reflect real world conditions, we introduce a new video understanding benchmark RiskCueBench in which videos are carefully annotated to identify a risk signal clip, defined as the earliest moment that indicates a potential safety concern. Experimental results reveal a significant gap in current systems ability to interpret evolving situations and anticipate future risky events from early visual signals, highlighting important challenges for deploying video risk prediction models in practice.

Figures (6)

• Figure 1: Many existing benchmarks ask about events which occur during the video. Our RiskCueBench focuses on predicting future risky events (details in §\ref{['sec:benchmark']}).
• Figure 2: Overview of our pipeline to curate risk signal clips from real-world incidents and evaluate VLM reasoning. Collection: We collect a large candidate set of YouTube videos using domain-relevant keywords. Scoring: The collected videos are filtered to only retain those with potential risk and high difficulty. Annotation: Human annotators label the filtered videos to identify the risk signal clip. Evaluation: Popular VLMs are presented with only the risk signal clip, and their output reasoning traces are evaluated.
• Figure 3: Distribution of Relatedness Score. Correct predictions (green) exhibit significantly higher mean relatedness scores than incorrect ones (red), indicating that accurate risk anticipation is strongly tied to better visual grounding.
• Figure 4: Reasoning Length Distribution. Incorrect predictions (red) are associated with significantly longer and more complex reasoning traces compared to correct ones (green), suggesting that model's overthinking or circular deliberation often leads to performance degradation.
• Figure 5: Taxonomy of VLM failure modes in situational risk assessment, categorized by perceptual, reasoning, and conclusion errors. Examples include perceptual errors (Type 1) miss critical cues like aggressive shoving, reasoning errors (Type 2) like failing to interpret normal vs. abnormal behaviors, and incorrect answers (Type 3).