Road Surface Condition Detection with Machine Learning using New York State Department of Transportation Camera Images and Weather Forecast Data

TL;DR

This work tackles automated road surface condition classification across New York State using NYSDOT camera images paired with HRRR weather forecasts. It introduces a two-branch Surface Condition Model (CNN for image features followed by a weather-informed Random Forest) and an Obstruction Detection Model (CNN), trained on a hand-labeled dataset of roughly 21–22k images spanning 45 camera sites across six classes. The study emphasizes generalizability to unseen cameras through site-specific nested cross-validation and demonstrates that incorporating weather data substantially improves predictive performance, achieving 81.5% accuracy on unseen sites, with obstruction recall exceeding 88%. The results support operational deployment by NYSDOT, highlighting the value of co-design, diverse site representation, and multimodal inputs for reliable, frequent road-condition monitoring during winter events.

Abstract

The New York State Department of Transportation (NYSDOT) has a network of roadside traffic cameras that are used by both the NYSDOT and the public to observe road conditions. The NYSDOT evaluates road conditions by driving on roads and observing live cameras, tasks which are labor-intensive but necessary for making critical operational decisions during winter weather events. However, machine learning models can provide additional support for the NYSDOT by automatically classifying current road conditions across the state. In this study, convolutional neural networks and random forests are trained on camera images and weather data to predict road surface conditions. Models are trained on a hand-labeled dataset of ~22,000 camera images, each classified by human labelers into one of six road surface conditions: severe snow, snow, wet, dry, poor visibility, or obstructed. Model generalizability is prioritized to meet the operational needs of the NYSDOT decision makers, and the weather-related road surface condition model in this study achieves an accuracy of 81.5% on completely unseen cameras.

Figures (6)

• Figure 1: Existing tool and motivation. (Left) Example of the 511ny.org map on February 6, 2025. Road surface conditions are color coded according to the key provided, with most of the state showing Snow/Ice conditions (pink). (Right) Example of one camera in Buffalo, NY on November 19, 2022, with two images taken five minutes apart, demonstrating how quickly snow can accumulate on the roads.
• Figure 2: Operational model flow and example. (Left) The Surface Condition Model is a two-stage process using a CNN and a RF. The Obstruction Detection Model solely predicts whether an image is obstructed of non-obstructed. (Right) The model flow in a practical inference scenario is shown, demonstrating that both predictions will be provided in conjunction for each new observation.
• Figure 3: Nested CV for the Surface Condition Model. For each test dataset (black cells), inner 5-fold CV is used on the remaining five folds (right), resulting in five models. The five models are used to create an ensemble prediction for the test dataset. This process is demonstrated for Test 1 and is repeated for each test set (left).
• Figure 4: Data splitting for the Obstruction Detection Model. Sampling, 2-fold CV, and ensembling for the two-class dataset is shown.
• Figure 5: Surface Condition Model: Validation Performance Comparison Across Models. (A) Validation accuracy of the 36 baseline CNN models are shown; this includes Architecture-Specific Top (greens), Generic Top (blues), and None (reds), and the inclusion of Augmentation (lighter shades), and No Augmentation (darker shades). (B) The top performing hyperparameter set for each of the five algorithms from the stage 2 downstream model are shown, including overall accuracy, as well as average recall metric.