A New Dataset and Framework for Robust Road Surface Classification via Camera-IMU Fusion

TL;DR

This work tackles the brittleness of road surface classification by introducing ROAD, a multimodal dataset with synchronized camera and IMU data across real-world, vision-only, and synthetic conditions. It couples a vision stream (EfficientNet-B0) with an inertial stream (CNN-BLSTM) and fuses them via bidirectional cross-attention and an adaptive gating mechanism to handle domain shifts and adverse conditions. Across PVS and ROAD, the proposed framework achieves state-of-the-art accuracy, with pronounced gains on minority surface classes and strong robustness under nighttime, rain, and surface transitions, while maintaining competitive performance in clean visual settings. ROAD enables systematic analysis of multimodal fusion robustness, sensor degradation, and out-of-distribution generalization, supporting practical deployment in diverse, cost-constrained environments.

Abstract

Road surface classification (RSC) is a key enabler for environment-aware predictive maintenance systems. However, existing RSC techniques often fail to generalize beyond narrow operational conditions due to limited sensing modalities and datasets that lack environmental diversity. This work addresses these limitations by introducing a multimodal framework that fuses images and inertial measurements using a lightweight bidirectional cross-attention module followed by an adaptive gating layer that adjusts modality contributions under domain shifts. Given the limitations of current benchmarks, especially regarding lack of variability, we introduce ROAD, a new dataset composed of three complementary subsets: (i) real-world multimodal recordings with RGB-IMU streams synchronized using a gold-standard industry datalogger, captured across diverse lighting, weather, and surface conditions; (ii) a large vision-only subset designed to assess robustness under adverse illumination and heterogeneous capture setups; and (iii) a synthetic subset generated to study out-of-distribution generalization in scenarios difficult to obtain in practice. Experiments show that our method achieves a +1.4 pp improvement over the previous state-of-the-art on the PVS benchmark and an +11.6 pp improvement on our multimodal ROAD subset, with consistently higher F1-scores on minority classes. The framework also demonstrates stable performance across challenging visual conditions, including nighttime, heavy rain, and mixed-surface transitions. These findings indicate that combining affordable camera and IMU sensors with multimodal attention mechanisms provides a scalable, robust foundation for road surface understanding, particularly relevant for regions where environmental variability and cost constraints limit the adoption of high-end sensing suites.

Figures (12)

• Figure 1: Overview of the proposed multimodal framework for road surface classification. An EfficientNet-B0 vision encoder processes raw RGB frames, while a CNN-BLSTM encoder processes inertial measurements (accelerometer and gyroscope). Each branch applies its own modality-specific LayerNorm+MLP tokenization module to generate visual and inertial tokens. These tokens are then aligned via a bidirectional cross-attention mechanism, in which each modality queries the other to exchange contextual information and produce refined representations $V'$ and $A'$. The resulting embeddings are aggregated and combined through an adaptive gating fusion module, which computes sample-dependent modality weights to produce a single fused representation $F$. Finally, $F$ is passed to a classification head to predict the road surface type.
• Figure 2: Examples of basic preprocessing operations applied to the input RGB frames. The top row shows the original images, while the bottom row illustrates simple transforms such as resizing, center cropping, and mild color/geometry adjustments, common on computer vision pipelines.
• Figure 3: Examples of data augmentation using the Automold library. The top row shows original images, and the bottom row displays augmented versions that simulate environmental effects such as brightness variation, rain, a dirty lens, and motion distortion.
• Figure 4: Representative samples from the proposed multimodal sensor fusion dataset, showcasing the three road surface classes available in our dataset: Asphalt, Belgian Blocks, and Off-road. Images were captured on public roads using a fixed, industry-standard camera mounting configuration.
• Figure 5: Schematic visualization of the placement of the five IMU units around the truck. The truck illustration was generated using a Generative AI tool for clarity and does not correspond to a specific vehicle model.