Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

JaywalkerVR: A VR System for Collecting Safety-Critical Pedestrian-Vehicle Interactions

Kenta Mukoya, Erica Weng, Rohan Choudhury, Kris Kitani

TL;DR

JaywalkerVR introduces a VR-based human-in-the-loop simulator built on CARLA to collect realistic, head-motion annotated pedestrian–vehicle interaction data in long-tail, safety-critical scenarios. The resulting CARLA-VR dataset enables pre-training and evaluation of trajectory forecasting models, improving robustness in interactive scenes. Using AgentFormer as a baseline, fine-tuning with CARLA-VR yields substantial gains in displacement error and collision-rate metrics, demonstrating practical benefits for safer autonomous driving. The work highlights the value of immersive VR data collection for reducing sim-to-real gap and enhancing model performance in rare but critical interactions.

Abstract

Developing autonomous vehicles that can safely interact with pedestrians requires large amounts of pedestrian and vehicle data in order to learn accurate pedestrian-vehicle interaction models. However, gathering data that include crucial but rare scenarios - such as pedestrians jaywalking into heavy traffic - can be costly and unsafe to collect. We propose a virtual reality human-in-the-loop simulator, JaywalkerVR, to obtain vehicle-pedestrian interaction data to address these challenges. Our system enables efficient, affordable, and safe collection of long-tail pedestrian-vehicle interaction data. Using our proposed simulator, we create a high-quality dataset with vehicle-pedestrian interaction data from safety critical scenarios called CARLA-VR. The CARLA-VR dataset addresses the lack of long-tail data samples in commonly used real world autonomous driving datasets. We demonstrate that models trained with CARLA-VR improve displacement error and collision rate by 10.7% and 4.9%, respectively, and are more robust in rare vehicle-pedestrian scenarios.

JaywalkerVR: A VR System for Collecting Safety-Critical Pedestrian-Vehicle Interactions

TL;DR

JaywalkerVR introduces a VR-based human-in-the-loop simulator built on CARLA to collect realistic, head-motion annotated pedestrian–vehicle interaction data in long-tail, safety-critical scenarios. The resulting CARLA-VR dataset enables pre-training and evaluation of trajectory forecasting models, improving robustness in interactive scenes. Using AgentFormer as a baseline, fine-tuning with CARLA-VR yields substantial gains in displacement error and collision-rate metrics, demonstrating practical benefits for safer autonomous driving. The work highlights the value of immersive VR data collection for reducing sim-to-real gap and enhancing model performance in rare but critical interactions.

Abstract

Developing autonomous vehicles that can safely interact with pedestrians requires large amounts of pedestrian and vehicle data in order to learn accurate pedestrian-vehicle interaction models. However, gathering data that include crucial but rare scenarios - such as pedestrians jaywalking into heavy traffic - can be costly and unsafe to collect. We propose a virtual reality human-in-the-loop simulator, JaywalkerVR, to obtain vehicle-pedestrian interaction data to address these challenges. Our system enables efficient, affordable, and safe collection of long-tail pedestrian-vehicle interaction data. Using our proposed simulator, we create a high-quality dataset with vehicle-pedestrian interaction data from safety critical scenarios called CARLA-VR. The CARLA-VR dataset addresses the lack of long-tail data samples in commonly used real world autonomous driving datasets. We demonstrate that models trained with CARLA-VR improve displacement error and collision rate by 10.7% and 4.9%, respectively, and are more robust in rare vehicle-pedestrian scenarios.
Paper Structure (21 sections, 4 figures, 1 table)

This paper contains 21 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Trajectory Forecasting
  4. Trajectory Datasets
  5. VR Pedestrian and Vehicle Simulators
  6. VR Simulator For Vehicle-Pedestrian Interaction Data Generation
  7. System Overview
  8. Walker Control $\&$ Tracking
  9. Pedestrian Model
  10. Scenario Generation $\&$ Data Recording
  11. Simulation Settings
  12. CARLA-VR Dataset
  13. Interactive Scenarios
  14. Dataset Details
  15. Experiments
  16. ...and 6 more sections

Figures (4)

  • Figure 1:
  • Figure 3: Room and equipment setting. (I) VIVE Pro 2: used for 1.visualizing simulator images and 2.tracking human position/rotation in the room (II) VIVE Wireless Adapter: used for allowing VIVE Pro 2 completely wireless (III) BaseStation 2.0: Track VR headset's position/rotation based on lighthouse tracking algorithm (IV) Simulator PC: Execute CARLA based VR simulator.
  • Figure 4: Experimental scenarios. (1) Jaywalk: Pedestrians jaywalk toward a bus stop while avoiding vehicles. (2) Parked Cars: Pedestrians avoid parked vehicles and move along the road. (3) 4-Way Stop: Pedestrians cross the crosswalk, avoiding cars at a four-way stop. (4) Parking Lot Entrance: Pedestrians walk through the entrance or a parking lot paying attention to cars.
  • Figure 5: Visualization result. Left figure: predicted trajectories of AgentFormer-B, Right figure: predicted trajectories of AgentFormer-VR. Light gray areas indicate the drivable area and dark gray areas indicate pedestrian crossings.