Behavioural gap assessment of human-vehicle interaction in real and virtual reality-based scenarios in autonomous driving

TL;DR

This study defines and quantifies the behavioural gap between real and VR pedestrian behavior in autonomous-driving contexts using a digital twin crosswalk implemented in CARLA. It combines real-world experiments with immersive VR, collecting objective metrics such as $T_{av}$, $CIT$, $L$, $F_p$, and $G$, along with subjective safety and presence assessments, across conditions with different eHMI states and braking styles. Results show that VR tends to elevate gaze toward the vehicle, slow movement, and alter crossing decisions, with explicit eHMI signaling and braking strategy affecting actions differently across real and virtual environments. The work provides a practical framework for validating VR-based studies against real-world behavior and highlights the need to calibrate VR cues to ensure faithful representations of pedestrian-AV interactions.

Abstract

In the field of autonomous driving research, the use of immersive virtual reality (VR) techniques is widespread to enable a variety of studies under safe and controlled conditions. However, this methodology is only valid and consistent if the conduct of participants in the simulated setting mirrors their actions in an actual environment. In this paper, we present a first and innovative approach to evaluating what we term the behavioural gap, a concept that captures the disparity in a participant's conduct when engaging in a VR experiment compared to an equivalent real-world situation. To this end, we developed a digital twin of a pre-existed crosswalk and carried out a field experiment (N=18) to investigate pedestrian-autonomous vehicle interaction in both real and simulated driving conditions. In the experiment, the pedestrian attempts to cross the road in the presence of different driving styles and an external Human-Machine Interface (eHMI). By combining survey-based and behavioural analysis methodologies, we develop a quantitative approach to empirically assess the behavioural gap, as a mechanism to validate data obtained from real subjects interacting in a simulated VR-based environment. Results show that participants are more cautious and curious in VR, affecting their speed and decisions, and that VR interfaces significantly influence their actions.

Figures (9)

• Figure 1: Digital twin for human-vehicle interaction in autonomous driving. (a) 3D crosswalk scenario. (b) Pedestrian attempting to cross. (c) Autonomous vehicle (eHMI, driving style). (d) Ambient sound, lighting and traffic signals. (e) Physical versus virtual sensors.
• Figure 2: Smooth (continuous line) and aggressive (dashed line) deceleration.
• Figure 3: AV with eHMI activated communicating green (a) and red (b) status. Virtual (above) and physical (bottom) design.
• Figure 4: Crossing decision event. (i) The pedestrian takes two steps forward to gain visibility. (ii) The vehicle is approaching and the pedestrian slows down without stopping. (iii) The pedestrian makes the decision to cross.
• Figure 5: Pedestrian-AV interaction in VR setup. (Upper row) The pedestrian waits while eHMI displays a red status. (Lower row) The eHMI switches to green status and the pedestrian decides to cross. From left to right: VR experimentation environment; overview of the simulated virtual scenario; pedestrian perspective; AV perspective (simulated camera).