Reliving the Dataset: Combining the Visualization of Road Users' Interactions with Scenario Reconstruction in Virtual Reality

TL;DR

Reliving the Dataset proposes a dual analysis framework that combines objective detection of critical traffic scenes via a semantic scene graph and multiple criticality measures with VR-based experiential analysis for deeper understanding. The approach uses an agnostic data interface and ROS-based OA module, along with a SA module that recreates scenes in CARLA in VR, enabling multiple perspectives and sensor augmentation. The key contributions are the integration of $TTC$, $RSS$, and $SFF$ measures within a semantic graph, the visualization tools for abstract and spatio-temporal views, and the VR recreation pipeline for knowledge discovery. This work advances dataset-driven development of HAD by enabling efficient corner-case detection and immersive scene exploration, with potential to extend to online data streams and VR-enabled data augmentation, including scenarios with vulnerable road users.

Abstract

One core challenge in the development of automated vehicles is their capability to deal with a multitude of complex trafficscenarios with many, hard to predict traffic participants. As part of the iterative development process, it is necessary to detect criticalscenarios and generate knowledge from them to improve the highly automated driving (HAD) function. In order to tackle this challenge,numerous datasets have been released in the past years, which act as the basis for the development and testing of such algorithms.Nevertheless, the remaining challenges are to find relevant scenes, such as safety-critical corner cases, in these datasets and tounderstand them completely.Therefore, this paper presents a methodology to process and analyze naturalistic motion datasets in two ways: On the one hand, ourapproach maps scenes of the datasets to a generic semantic scene graph which allows for a high-level and objective analysis. Here,arbitrary criticality measures, e.g. TTC, RSS or SFF, can be set to automatically detect critical scenarios between traffic participants.On the other hand, the scenarios are recreated in a realistic virtual reality (VR) environment, which allows for a subjective close-upanalysis from multiple, interactive perspectives.

Figures (8)

• Figure 1: A critical scene that was first identified from a naturalistic motion dataset and later reconstructed in our simulation environment. The reconstruction allows for scenario exploration in virtual reality, where the user has full control over the time dimension and can relive the scenario from all possible points of view.
• Figure 2: Overview of the framework for visual analysis. Using naturalistic motion datasets, we apply both an objective and a subjective analysis. The objective analysis module is used to find critical scenes within the dataset using different metrics and provides abstract visualisation options for scenes and their surrounding scenarios. The subjective analysis module allows for a reconstruction and an in-depth investigation of these scenes in a VR environment.
• Figure 3: Abstract visualizations from the OA module. In (a), the scene graph described in \ref{['sec:concept:scene_graph']} is being visualized. Traffic participants are displayed as colored boxes. Relations between them, built up by the scene graph, are displayed as white lines and colored spheres, which represent their criticality status. Darker spheres represent more critical relations. In (b), the spatio-temporal visualization of a scenario is shown. This enables a quick understanding of the scenario.
• Figure 4: Workflow for the recreation of scenarios from datasets in a 3D simulation environment. Using the dataset and data from OpenStreetMap, both the static and dynamic environment can be reconstructed.
• Figure 5: The virtual reality setup in our laboratory. The setup provides a walking area in order to move within the recreated traffic constellation.