RiskBench: A Scenario-based Benchmark for Risk Identification

TL;DR

RiskBench tackles risk identification for intelligent driving systems using a scenario-based benchmark built in CARLA. It establishes a four-type scenario taxonomy, a data collection and augmentation pipeline, and a three-metric evaluation (risk localization, risk anticipation, and planning awareness) including a novel Influenced Ratio to link identification accuracy to planning outcomes. The work benchmarks ten risk-identification algorithms, revealing tradeoffs between perception-driven and prediction-driven methods and highlighting temporal consistency and planning integration as key challenges. By offering a standardized dataset and evaluation protocol, RiskBench aims to accelerate robust, planning-aware risk identification toward safer autonomous driving deployments.

Abstract

Intelligent driving systems aim to achieve a zero-collision mobility experience, requiring interdisciplinary efforts to enhance safety performance. This work focuses on risk identification, the process of identifying and analyzing risks stemming from dynamic traffic participants and unexpected events. While significant advances have been made in the community, the current evaluation of different risk identification algorithms uses independent datasets, leading to difficulty in direct comparison and hindering collective progress toward safety performance enhancement. To address this limitation, we introduce \textbf{RiskBench}, a large-scale scenario-based benchmark for risk identification. We design a scenario taxonomy and augmentation pipeline to enable a systematic collection of ground truth risks under diverse scenarios. We assess the ability of ten algorithms to (1) detect and locate risks, (2) anticipate risks, and (3) facilitate decision-making. We conduct extensive experiments and summarize future research on risk identification. Our aim is to encourage collaborative endeavors in achieving a society with zero collisions. We have made our dataset and benchmark toolkit publicly on the project page: https://hcis-lab.github.io/RiskBench/

Figures (4)

• Figure 1: Scenario-based Evaluation for Risk Identification. We establish four distinct interaction types to cover various risk definitions explored in the community. RiskBench evaluates an algorithm's ability to identify risks stemming from dynamic traffic participants and unexpected events based on localization, anticipation, and planning awareness.
• Figure 2: Scenario Collection Pipeline. The scenario taxonomy is designed to enable a systematic collection of ground truth risks induced by dynamic traffic participants and unexpected events. The taxonomy includes various attributes such as road topology, scenario types, ego vehicle behavior, and traffic participants' behavior. From this taxonomy, if a scenario script is set, two human subjects can act accordingly. To form the final scenario dataset, we augment the collected scenario by changing attributes, including time of day, weather conditions, and traffic density.
• Figure 3: The RiskBench Dataset Statistics. We denote Crosswalking as CW, Jay-walking as JW, Left Lane Change as LLC, Right Lane Change as RLC, Go into Roundabout as GIR, Exit Roundabout as ER, and Go Around Roundabout as GAR.
• Figure 4: Planning-aware Metric. An ideal planner should be able to yield to the crossing vehicle as shown in (a). If a risk identification model successfully identifies the vehicle as the risk, we hide all the other objects. The planner should plan a slow-down path, as shown in (b). In contrast, if the model identifies the wrong object as risk (e.g., the green one), the planner will plan a path to (nearly) collide with the True Risk, as shown in (c).