DOLPHINS: Dataset for Collaborative Perception enabled Harmonious and Interconnected Self-driving

TL;DR

The experiment results show that the raw-level fusion scheme through V2X communication can help to improve the precision as well as to reduce the necessity of expensive LiDAR equipment on vehicles when RSUs exist, which may accelerate the popularity of interconnected self-driving vehicles.

Abstract

Vehicle-to-Everything (V2X) network has enabled collaborative perception in autonomous driving, which is a promising solution to the fundamental defect of stand-alone intelligence including blind zones and long-range perception. However, the lack of datasets has severely blocked the development of collaborative perception algorithms. In this work, we release DOLPHINS: Dataset for cOllaborative Perception enabled Harmonious and INterconnected Self-driving, as a new simulated large-scale various-scenario multi-view multi-modality autonomous driving dataset, which provides a ground-breaking benchmark platform for interconnected autonomous driving. DOLPHINS outperforms current datasets in six dimensions: temporally-aligned images and point clouds from both vehicles and Road Side Units (RSUs) enabling both Vehicle-to-Vehicle (V2V) and Vehicle-to-Infrastructure (V2I) based collaborative perception; 6 typical scenarios with dynamic weather conditions make the most various interconnected autonomous driving dataset; meticulously selected viewpoints providing full coverage of the key areas and every object; 42376 frames and 292549 objects, as well as the corresponding 3D annotations, geo-positions, and calibrations, compose the largest dataset for collaborative perception; Full-HD images and 64-line LiDARs construct high-resolution data with sufficient details; well-organized APIs and open-source codes ensure the extensibility of DOLPHINS. We also construct a benchmark of 2D detection, 3D detection, and multi-view collaborative perception tasks on DOLPHINS. The experiment results show that the raw-level fusion scheme through V2X communication can help to improve the precision as well as to reduce the necessity of expensive LiDAR equipment on vehicles when RSUs exist, which may accelerate the popularity of interconnected self-driving vehicles. DOLPHINS is now available on https://dolphins-dataset.net/.

Figures (6)

• Figure 1: An illustration of the advantages of collaborative perception over stand-alone intelligence
• Figure 2: An example of multi-view object detection in DOLPHINS dataset. There is a right merging lane in front of the ego vehicle. Because of the occlusion, the ego vehicle can hardly detect the purple vehicle ( red box) on the branch and the police car ( blue box). The auxiliary vehicle is in front of the ego vehicle, which can see both object vehicles distinctly. Additionally, the RSU can detect another two vehicles ( purple box) on the branch
• Figure 3: A comparison with 3 brand new collaborative perception datasets: OPV2V opv2v, V2X-Sim v2x-sim, and DAIR-V2X-C yu2022dair, as well as 2 well-known single-vehicle autonomous driving datasets: KITTI Geiger2012CVPR and nuScenes caesar2020nuscenes. A detailed comparison is provided in Sec. \ref{['sec:related']}
• Figure 4: An illustration of temporary-aligned images and point clouds from three viewpoints. The position of each viewpoint is demonstrated in Fig. \ref{['fig:sample']}
• Figure 5: All ego vehicles are driving along a pre-defined route ( green arrows), while each RSU camera is settled with a fixed direction and range ( blue or brown sector mark). We also mark positions where the ego vehicle or possible auxiliary vehicles are initialized. Among all scenarios, (a) and (e) are two intersection scenarios; (b) is the scenario of a T-junction with moderate rain; (c) is also a crossroads while the ego vehicle is on a steep ramp; (d) is a scenario existing a right merging lane on the expressway, and the weather is foggy; (f) is the scenario of a mountain road. All scenarios have plenty of occlusion situations