HetroD: A High-Fidelity Drone Dataset and Benchmark for Autonomous Driving in Heterogeneous Traffic

TL;DR

HetroD introduces the first drone-view dataset and benchmark focused on heterogeneous traffic with VRUs, combining centimeter-accurate annotations, HD maps, and traffic signals across $17.5$ hours and $65.4k$ trajectories ($70\%$ VRUs). The authors provide a unified toolkit to enable forecasting, planning, and simulation across datasets, and demonstrate that state-of-the-art methods struggle with lateral VRU movements and unstructured maneuvers in dense multi-agent scenes. Through cross-dataset forecasting, scenario-conditioned evaluation, and latent-embedding analyses, the work reveals significant generalization gaps and the strong influence of viewpoint and map on performance. The results highlight two main failure modes—two-wheelers are particularly hard to predict, and rule-based planners exhibit high lateral VRU collision rates—motivating VRU-aware modeling and simulation to improve safety in real-world heterogeneous traffic.

Abstract

We present HetroD, a dataset and benchmark for developing autonomous driving systems in heterogeneous environments. HetroD targets the critical challenge of navi- gating real-world heterogeneous traffic dominated by vulner- able road users (VRUs), including pedestrians, cyclists, and motorcyclists that interact with vehicles. These mixed agent types exhibit complex behaviors such as hook turns, lane splitting, and informal right-of-way negotiation. Such behaviors pose significant challenges for autonomous vehicles but remain underrepresented in existing datasets focused on structured, lane-disciplined traffic. To bridge the gap, we collect a large- scale drone-based dataset to provide a holistic observation of traffic scenes with centimeter-accurate annotations, HD maps, and traffic signal states. We further develop a modular toolkit for extracting per-agent scenarios to support downstream task development. In total, the dataset comprises over 65.4k high- fidelity agent trajectories, 70% of which are from VRUs. HetroD supports modeling of VRU behaviors in dense, het- erogeneous traffic and provides standardized benchmarks for forecasting, planning, and simulation tasks. Evaluation results reveal that state-of-the-art prediction and planning models struggle with the challenges presented by our dataset: they fail to predict lateral VRU movements, cannot handle unstructured maneuvers, and exhibit limited performance in dense and multi-agent scenarios, highlighting the need for more robust approaches to heterogeneous traffic. See our project page for more examples: https://hetroddata.github.io/HetroD/

Figures (7)

• Figure 1: HetroD is a high-fidelity, drone dataset that captures unstructured maneuvers such as hook turns, aggressive overtakes, queue cutting, and congested crossings among vehicles, scooters, and pedestrians in heterogeneous traffic environments. These maneuvers are critical for testing autonomous driving systems yet remain underexplored in the community. To address this, we construct a benchmark to evaluate existing methods in motion planning, motion prediction, traffic simulation, and conduct a thorough investigation of their generalization across datasets.
• Figure 2: Aerial views of the six recording locations in HetroD, capturing diverse urban traffic scenarios.
• Figure 3: Agent-type distribution of HetroD (center) compared against two prior datasets, inD and SinD.
• Figure 4: Cross-type interaction patterns and TTC/DRAC distributions. In the chord diagrams (top), link thickness indicates the number of cross-type interactions between agent categories, with color denoting time-to-collision (TTC) ttc bands (0–1, 1–2, 2–3, 3–4, 4–5 s; lower indicates higher risk). The bottom panels show marginal distributions of TTC and deceleration rate to avoid crash (DRAC) drac. HetroD exhibits denser and riskier cross-type interactions, particularly among vehicles, two-wheelers, and pedestrians with a clear shift toward shorter TTC and higher DRAC, highlighting the prevalence of complex VRU interactions in HetroD compared to other datasets.
• Figure 5: We develop a unified development toolkit that converts a wide range of traffic scene datasets into standardized, agent-centric data formats scenarionettrajdatacharraut2025vmaxreinforcementlearningframeworkgpudrive, enabling seamless comparisons across datasets for forecasting, planning, and simulation.