Real-Time Environment Condition Classification for Autonomous Vehicles

TL;DR

This work tackles the challenge of real-time environment condition assessment for autonomous driving beyond geo-fenced limits by introducing RECNet, a lightweight RGB-frame classifier built on EfficientNet-B2. It couples a novel DENSE++ label taxonomy with a semi-automated labeling pipeline (including LiDAR-based precipitation intensity) to enable joint classification of daytime, precipitation, fog, road/sidewalk conditions, and scene-setting at 20 Hz. The approach achieves a high overall accuracy (~92% under hierarchical labeling) and demonstrates strong per-class performance, with interpretable Grad-CAM visualizations confirming sensible model focus. The results support safer, dynamically available autonomous driving capabilities and pave the way for multi-modal, temporally-aware conditioning in future work.

Abstract

Current autonomous driving technologies are being rolled out in geo-fenced areas with well-defined operation conditions such as time of operation, area, weather conditions and road conditions. In this way, challenging conditions as adverse weather, slippery road or densely-populated city centers can be excluded. In order to lift the geo-fenced restriction and allow a more dynamic availability of autonomous driving functions, it is necessary for the vehicle to autonomously perform an environment condition assessment in real time to identify when the system cannot operate safely and either stop operation or require the resting passenger to take control. In particular, adverse-weather challenges are a fundamental limitation as sensor performance degenerates quickly, prohibiting the use of sensors such as cameras to locate and monitor road signs, pedestrians or other vehicles. To address this issue, we train a deep learning model to identify outdoor weather and dangerous road conditions, enabling a quick reaction to new situations and environments. We achieve this by introducing an improved taxonomy and label hierarchy for a state-of-the-art adverse-weather dataset, relabelling it with a novel semi-automated labeling pipeline. Using the novel proposed dataset and hierarchy, we train RECNet, a deep learning model for the classification of environment conditions from a single RGB frame. We outperform baseline models by relative 16% in F1- Score, while maintaining a real-time capable performance of 20 Hz.

Figures (7)

• Figure 1: We introduce RECNet, a unified model for real-time classification of environment conditions from a single RGB frame. This is enabled by our novel taxonomy and our proposed semi-automated labeling pipeline.
• Figure 2: Distribution of the refined and previous labels for the overarching categories: daytime, weather, and road conditions. The additional "twilight" tag clearly separates day from night. Precipitation is defined such that it can occur simultaneously with fog, leading to improved scene descriptions. Finally, separating road and sidewalk conditions improves accuracy, particularly for scenarios like dry roads with snow-covered sidewalks, which were previously grouped together.
• Figure 3: Precipitation Intensity Annotation. Lidar points without a sufficient number of neighborhood points are classified as stemming from precipitation, as visualized for a heavy snow scene (right). Upper and lower row show point clouds without and with clutter points removed.
• Figure 4: Environment classification hierarchy. The newly introduced label hierarchy allows to assign multiple weather conditions to a single frame, introduces a classification of the precipitation intensity, and differentiates between road and sidewalk.
• Figure 5: Network overview figure for the proposed RECNet. As the backbone, we use the EfficientNet-B2 network, a lightweight deep learning model capable of extracting a general feature map. This map is utilized by the six downstream classification units to classify all annotated environmental conditions incl. Fog, Daytime, Road Conditions, Precipitation and Infrastructure.