AVOID: The Adverse Visual Conditions Dataset with Obstacles for Driving Scene Understanding

TL;DR

AVOID addresses the lack of obstacle-focused datasets under adverse visual conditions by introducing a large-scale, multi-modal synthetic dataset in the CARLA simulator with synchronized RGB, depth, LiDAR, BEV maps, and waypoint data. It enables obstacle detection, semantic segmentation, depth estimation, and waypoint prediction across 42 weather-daytime scenarios and eight Town routes, with train/validation/test splits including unseen obstacles. The authors benchmark transformer-based fusion networks (SwiftFuser) against baselines (RODSNet, TransFuser) and conduct ablations on multi-task learning, achieving competitive obstacle IoU (up to 51.81% with a ResNet-18 backbone at 18.87 FPS) and near-perfect waypoint navigation on test sets. This work provides a practical resource for robust driving perception in adverse conditions and demonstrates the value of richly annotated synthetic data for joint perception and navigation tasks.

Abstract

Understanding road scenes for visual perception remains crucial for intelligent self-driving cars. In particular, it is desirable to detect unexpected small road hazards reliably in real-time, especially under varying adverse conditions (e.g., weather and daylight). However, existing road driving datasets provide large-scale images acquired in either normal or adverse scenarios only, and often do not contain the road obstacles captured in the same visual domain as for the other classes. To address this, we introduce a new dataset called AVOID, the Adverse Visual Conditions Dataset, for real-time obstacle detection collected in a simulated environment. AVOID consists of a large set of unexpected road obstacles located along each path captured under various weather and time conditions. Each image is coupled with the corresponding semantic and depth maps, raw and semantic LiDAR data, and waypoints, thereby supporting most visual perception tasks. We benchmark the results on high-performing real-time networks for the obstacle detection task, and also propose and conduct ablation studies using a comprehensive multi-task network for semantic segmentation, depth and waypoint prediction tasks.

Figures (6)

• Figure 1: Sample views of all obstacles in AVOID. AVOID provides a set of multi-view images and annotations for each observed driving scene. As the vehicle navigates through the simulated routes, it collects a stereo pair of egocentric RGB images (top row) and the corresponding annotations (semantic and depth maps, RGB and semantic maps in Bird's-Eye-View (BEV), and raw and semantic LiDAR data, shown from left to right on bottom two rows)
• Figure 2: Samples results of the vehicle avoiding obstacles ahead (a nightstand, a black suitcase, and a black cow, from top to bottom in order).
• Figure 3: Comparison of data distributions for pixel-wise semantic labels across the segmentation datasets.
• Figure 4: Town routes (Train: 1--4, Val.: 5, 6, Test: 7, 10) in the CARLA Simulator. Note that only one sample from each town is shown.
• Figure 5: Overall network architecture of our multi-modal (RGB-D) transformer-based fusion network for multi-task learning, called SwiftFuser. We compare this network to RODSNet with GRU added for waypoint prediction, in which initial semantic and disparity maps are each averaged pooled and input to the GRU modules.