The ADUULM-360 Dataset -- A Multi-Modal Dataset for Depth Estimation in Adverse Weather

TL;DR

The ADUULM-360 dataset is presented, a novel multi-modal dataset for depth estimation that covers all established autonomous driving sensor modalities, cameras, lidars, and radars and is also the first depth estimation dataset that contains diverse scenes in good and adverse weather conditions.

Abstract

Depth estimation is an essential task toward full scene understanding since it allows the projection of rich semantic information captured by cameras into 3D space. While the field has gained much attention recently, datasets for depth estimation lack scene diversity or sensor modalities. This work presents the ADUULM-360 dataset, a novel multi-modal dataset for depth estimation. The ADUULM-360 dataset covers all established autonomous driving sensor modalities, cameras, lidars, and radars. It covers a frontal-facing stereo setup, six surround cameras covering the full 360-degree, two high-resolution long-range lidar sensors, and five long-range radar sensors. It is also the first depth estimation dataset that contains diverse scenes in good and adverse weather conditions. We conduct extensive experiments using state-of-the-art self-supervised depth estimation methods under different training tasks, such as monocular training, stereo training, and full surround training. Discussing these results, we demonstrate common limitations of state-of-the-art methods, especially in adverse weather conditions, which hopefully will inspire future research in this area. Our dataset, development kit, and trained baselines are available at https://github.com/uulm-mrm/aduulm_360_dataset.

Figures (4)

• Figure 1: An example from the ADUULM-360 dataset in light rain conditions. The top shows lidar projections in the six surround camera views, the bottom left in the stereo views, and the bottom right shows the lidar and radar point clouds.
• Figure 2: Dataset samples of the frontal facing left stereo camera at different lighting and weather conditions. The first columns show examples of good weather; the middle two columns show examples of lighter and heavier rain. The last column shows examples of snowing conditions.
• Figure 3: Overview of the sensor setup used for the ADUULM-360 dataset. Camera sensors are depicted in red, lidar sensors in green, and radar sensors in blue.
• Figure 4: Depth predictions from MonoViT zhao2022monovit trained using monocular video sequences for different test splits of the ADUULM-360 dataset. The model generally struggles with dynamic objects moving at the same speed as the camera, as shown in (a). While light rain, as shown in (b), does not affect the results significantly, the model fails for more severe conditions such as snowfall shown in (c).