OmniHD-Scenes: A Next-Generation Multimodal Dataset for Autonomous Driving

TL;DR

OmniHD-Scenes addresses the need for a large-scale, multimodal autonomous-driving dataset with omnidirectional sensor coverage by integrating 128-beam LiDAR, six cameras, and six 4D imaging radars across 1,501 clips (≈450k frames, ≈5.85M sensor points). It introduces a novel 4D annotation pipeline and a dense 3D occupancy ground-truth generation method, enabling robust benchmarking for 3D object detection and semantic occupancy under varied conditions, including closed-test-site data. The paper provides comprehensive baselines across LiDAR, camera, radar, and fusion modalities, demonstrating the strengths and limitations of each approach, particularly highlighting the robustness of 4D radar in adverse weather and the benefits of sensor fusion. These contributions establish a new resource for cost-effective perception research and offer benchmarks that can drive development of robust perception algorithms for real-world autonomous driving, especially in challenging environments.

Abstract

The rapid advancement of deep learning has intensified the need for comprehensive data for use by autonomous driving algorithms. High-quality datasets are crucial for the development of effective data-driven autonomous driving solutions. Next-generation autonomous driving datasets must be multimodal, incorporating data from advanced sensors that feature extensive data coverage, detailed annotations, and diverse scene representation. To address this need, we present OmniHD-Scenes, a large-scale multimodal dataset that provides comprehensive omnidirectional high-definition data. The OmniHD-Scenes dataset combines data from 128-beam LiDAR, six cameras, and six 4D imaging radar systems to achieve full environmental perception. The dataset comprises 1501 clips, each approximately 30-s long, totaling more than 450K synchronized frames and more than 5.85 million synchronized sensor data points. We also propose a novel 4D annotation pipeline. To date, we have annotated 200 clips with more than 514K precise 3D bounding boxes. These clips also include semantic segmentation annotations for static scene elements. Additionally, we introduce a novel automated pipeline for generation of the dense occupancy ground truth, which effectively leverages information from non-key frames. Alongside the proposed dataset, we establish comprehensive evaluation metrics, baseline models, and benchmarks for 3D detection and semantic occupancy prediction. These benchmarks utilize surround-view cameras and 4D imaging radar to explore cost-effective sensor solutions for autonomous driving applications. Extensive experiments demonstrate the effectiveness of our low-cost sensor configuration and its robustness under adverse conditions. Data will be released at https://www.2077ai.com/OmniHD-Scenes.

Figures (15)

• Figure 1: Streamlined closed-loop data processing pipeline, primarily comprising data collection, cleaning, annotation, model training, validation, and deployment stages.
• Figure 2: Sensor deployment architecture diagram. Multiple views of the vehicle platform are provided to present a comprehensive visualization of the sensor placement.
• Figure 3: Sensor coordinate system. The red, green, and blue arrows represent the x- y-, and z-axes, respectively. The origin of the vehicle coordinate system is located at the center of the rear axle.
• Figure 4: Sensor calibration. The process primarily involves calibration of the intrinsic parameters of the cameras and the extrinsic parameters of the cameras, LiDAR, 4D radar, and GNSS.
• Figure 5: Time synchronization system topology. The GEACX2 domain controller acts as a PTP master clock, aligning with the GPS coordinated universal time (UTC). Other devices act as slave nodes to maintain the same time domain.