Sky-Drive: A Distributed Multi-Agent Simulation Platform for Human-AI Collaborative and Socially-Aware Future Transportation

TL;DR

Sky-Drive tackles the need for socially aware, human-centered autonomous transportation by introducing a distributed multi-agent simulation platform with a digital twin and a multi-modal human-in-the-loop. It adds a bidirectional human-AI collaboration mechanism (HAIM and AIHM) and demonstrates advanced use cases, including VR-based AV-HRU interactions, HAIM-DRL, VLM-RL, personalized curriculum learning, and accident data replay. The framework plans foundation-model integration and hardware-in-the-loop testing to bridge simulators and real-world deployments, enabling scenario generation, data collection, policy training, and hardware validation within a unified, open-source platform. If realized, Sky-Drive could become a foundational tool for developing socially-aware, human-aligned autonomous transportation systems.

Abstract

Recent advances in autonomous system simulation platforms have significantly enhanced the safe and scalable testing of driving policies. However, existing simulators do not yet fully meet the needs of future transportation research-particularly in enabling effective human-AI collaboration and modeling socially-aware driving agents. This paper introduces Sky-Drive, a novel distributed multi-agent simulation platform that addresses these limitations through four key innovations: (a) a distributed architecture for synchronized simulation across multiple terminals; (b) a multi-modal human-in-the-loop framework integrating diverse sensors to collect rich behavioral data; (c) a human-AI collaboration mechanism supporting continuous and adaptive knowledge exchange; and (d) a digital twin framework for constructing high-fidelity virtual replicas of real-world transportation environments. Sky-Drive supports diverse applications such as autonomous vehicle-human road users interaction modeling, human-in-the-loop training, socially-aware reinforcement learning, personalized driving development, and customized scenario generation. Future extensions will incorporate foundation models for context-aware decision support and hardware-in-the-loop testing for real-world validation. By bridging scenario generation, data collection, algorithm training, and hardware integration, Sky-Drive has the potential to become a foundational platform for the next generation of human-centered and socially-aware autonomous transportation systems research. The demo video and code are available at:https://sky-lab-uw.github.io/Sky-Drive-website/

Figures (7)

• Figure 1: Overview of Sky-Drive's key components and functionalities. (a) a distributed multi-agent architecture enabling synchronized simulation across multiple terminals; (b) a multi-modal human-in-the-loop framework capturing comprehensive behavioral data through integrated sensor systems; (c) a digital twin framework that creates high-fidelity virtual replicas of transportation systems through multi-source data integration; (d) a human-AI collaboration mechanism facilitating knowledge exchange between humans and AI systems; (e) the planned integration of foundation models to enhance decision-making, enabling more adaptive and context-aware human-AI collaboration; (f) a hardware-in-the-loop framework, planned for future integration, enabling remote control and data collection and ensuring that algorithms are evaluated in real-world environments.
• Figure 2: Workflow of Sky-Drive. (a) scenario generation & data collection through CARLA-based synthetic environments and digital twin integration of real-world traffic data; (b) simulation & algorithm training enabled by distributed multi-agent architecture and human-AI collaboration mechanism; (c) hardware integration & testing utilizing ROS compatibility for direct validation of autonomous driving algorithms on physical platforms.
• Figure 3: Illustration of Sky-Drive's distributed multi-agent architecture. Sky-Drive enables synchronized simulation across multiple terminals while maintaining precise real-time interactions between AVs, HVs, and pedestrians through RPC and Socket.IO-based communication platform, supporting comprehensive data collection and real-time analysis of multi-agent behaviors.
• Figure 4: VR-based experimental setup for studying AV-HRU interactions at unsignalized intersections.
• Figure 5: Qualitative examples. Each scenario is downsampled to four frames for visualisation.