SynSHRP2: A Synthetic Multimodal Benchmark for Driving Safety-critical Events Derived from Real-world Driving Data

TL;DR

SynSHRP2 tackles privacy and data scarcity in driving safety research by delivering a publicly available synthetic multimodal dataset derived from the SHRP 2 NDS. It employs a Stable Diffusion + ControlNet workflow to de-identify keyframes while preserving critical safety context, producing 1,874 crashes and 6,924 near-crashes with 5-second pre- and during-event time-series and rich narratives. The work presents two benchmarks—SCE attribute detection and SCE scene understanding—demonstrating the viability of synthetic data for standardized evaluation and ADS/ADAS development. By providing synchronized synthetic keyframes and narratives, SynSHRP2 enables robust benchmarking and cross-study comparability, while maintaining privacy protections. The approach also outlines a practical pipeline for generating privacy-preserving, high-fidelity multimodal driving data from real-world sources.

Abstract

Driving-related safety-critical events (SCEs), including crashes and near-crashes, provide essential insights for the development and safety evaluation of automated driving systems. However, two major challenges limit their accessibility: the rarity of SCEs and the presence of sensitive privacy information in the data. The Second Strategic Highway Research Program (SHRP 2) Naturalistic Driving Study (NDS), the largest NDS to date, collected millions of hours of multimodal, high-resolution, high-frequency driving data from thousands of participants, capturing thousands of SCEs. While this dataset is invaluable for safety research, privacy concerns and data use restrictions significantly limit public access to the raw data. To address these challenges, we introduce SynSHRP2, a publicly available, synthetic, multimodal driving dataset containing over 1874 crashes and 6924 near-crashes derived from the SHRP 2 NDS. The dataset features de-identified keyframes generated using Stable Diffusion and ControlNet, ensuring the preservation of critical safety-related information while eliminating personally identifiable data. Additionally, SynSHRP2 includes detailed annotations on SCE type, environmental and traffic conditions, and time-series kinematic data spanning 5 seconds before and during each event. Synchronized keyframes and narrative descriptions further enhance its usability. This paper presents two benchmarks for event attribute classification and scene understanding, demonstrating the potential applications of SynSHRP2 in advancing safety research and automated driving system development.

Figures (5)

• Figure 1: Example illustrating all data types in SynSHRP2.
• Figure 2: Workflow of the de-identified keyframe synthesis process. The pipeline begins with a keyframe input, applying super-resolution to upscale image resolution. Semantic segmentation then identifies object-related pixels, while Canny edge detection is used for objects where orientation is critical. The new frame is synthesized part by part by masking corresponding segments. Image cropping removes unnecessary elements like timestamps. For the "Impact" keyframe, this completes the process. For subsequent keyframes, an IP-adapter utilizes the first keyframe as an image prompt to ensure consistency across frames.
• Figure 3: Comparison of synthetic image generation approaches.
• Figure 4: Comparison between synthetic images with and without Canny.
• Figure 5: Comparison between synthetic images with and without IP-Adapter.