DeepUrban: Interaction-Aware Trajectory Prediction and Planning for Automated Driving by Aerial Imagery

TL;DR

DeepUrban addresses the lack of dense-urban interaction benchmarks by providing a drone-derived trajectory dataset with rich 3D traffic objects and open map data. The authors evaluate state-of-the-art prediction/planning methods (notably ScePT) on DeepUrban and demonstrate that incorporating the dataset into nuScenes yields substantial improvements, with up to $ADE$ improvements of about 44% and $FDE$ improvements of about 44% in cross-dataset tests. Generalization assessments across areas and datasets show that multi-location training and cross-domain transfer enhance predictive accuracy and safety metrics. The work is integrated into the TrajData dataloader to streamline benchmarking and includes plans for online evaluation and simulation-based testing (Carla). Overall, DeepUrban advances interaction-aware trajectory prediction and planning in dense urban settings and provides a scalable path toward broader benchmarking across datasets.

Abstract

The efficacy of autonomous driving systems hinges critically on robust prediction and planning capabilities. However, current benchmarks are impeded by a notable scarcity of scenarios featuring dense traffic, which is essential for understanding and modeling complex interactions among road users. To address this gap, we collaborated with our industrial partner, DeepScenario, to develop DeepUrban-a new drone dataset designed to enhance trajectory prediction and planning benchmarks focusing on dense urban settings. DeepUrban provides a rich collection of 3D traffic objects, extracted from high-resolution images captured over urban intersections at approximately 100 meters altitude. The dataset is further enriched with comprehensive map and scene information to support advanced modeling and simulation tasks. We evaluate state-of-the-art (SOTA) prediction and planning methods, and conducted experiments on generalization capabilities. Our findings demonstrate that adding DeepUrban to nuScenes can boost the accuracy of vehicle predictions and planning, achieving improvements up to 44.1 % / 44.3% on the ADE / FDE metrics. Website: https://iv.ee.hm.edu/deepurban

Figures (3)

• Figure 1: DeepUrban dataset: Data pipeline showing detected and tracked agents including 14 different agent types (top left), generated scenario data from raw inputs (top right) and predicted / planned clique agents using the ScePT algorithm ScePT2022 (bottom).
• Figure 2: Qualitative results on validation scenarios using ScePT: Scenario visualizations show the ego vehicle (pink), pedestrians (orange), and vehicles (red) along with prediction (solid lines), and GT trajectories (dashed lines). We highlight the adaptation of the planner as the ego interacts with accurately predicted pedestrians (first row), precise prediction but with planning errors (second row), dense traffic interactions with slow moving vehicles on the side (third row), and ego-vehicle overtaking, and interacting with pedestrians (last row).
• Figure 3: Four scenarios validated on nuScenes: Trained on nuScenes(N) (left) and trained on nuScenes + DeepUrban (ND) (right). Scenario visualizations show the ego vehicle (pink), pedestrians (orange), and vehicles (red) along with prediction (solid lines), and GT trajectories (dashed lines). We show: Biggest differences in ADE (here also biggest in FDE) with one mode / three modes (upper left pair / upper right pair). When trained on nuScenes only, vehicle predictions seem to be less conservative, resulting in bigger FDE and ADE; Biggest difference in collision score with one mode / three modes (lower left pair / lower right pair). When trained on nuScenes and DeepUrban, predictions seem to be more passive resulting in fewer collisions.