RASMD: RGB And SWIR Multispectral Driving Dataset for Robust Perception in Adverse Conditions

TL;DR

RASMD tackles the lack of public SWIR data for autonomous driving by introducing a large-scale RGB-SWIR multispectral dataset with 100K synchronized pairs and diverse conditions. It enables two benchmarks: object detection with modality-specific and merged test sets, and RGB→SWIR image translation to explore cross-domain generation. Experiments show that fusing RGB and SWIR improves detection performance, especially for pedestrians and other vulnerable road users in adverse weather, while translation methods indicate feasible generation of SWIR content from RGB and potential for data scale-up. This dataset is poised to advance robust, multispectral perception research in autonomous driving and underlines SWIR’s practical value in challenging environments.

Abstract

Current autonomous driving algorithms heavily rely on the visible spectrum, which is prone to performance degradation in adverse conditions like fog, rain, snow, glare, and high contrast. Although other spectral bands like near-infrared (NIR) and long-wave infrared (LWIR) can enhance vision perception in such situations, they have limitations and lack large-scale datasets and benchmarks. Short-wave infrared (SWIR) imaging offers several advantages over NIR and LWIR. However, no publicly available large-scale datasets currently incorporate SWIR data for autonomous driving. To address this gap, we introduce the RGB and SWIR Multispectral Driving (RASMD) dataset, which comprises 100,000 synchronized and spatially aligned RGB-SWIR image pairs collected across diverse locations, lighting, and weather conditions. In addition, we provide a subset for RGB-SWIR translation and object detection annotations for a subset of challenging traffic scenarios to demonstrate the utility of SWIR imaging through experiments on both object detection and RGB-to-SWIR image translation. Our experiments show that combining RGB and SWIR data in an ensemble framework significantly improves detection accuracy compared to RGB-only approaches, particularly in conditions where visible-spectrum sensors struggle. We anticipate that the RASMD dataset will advance research in multispectral imaging for autonomous driving and robust perception systems.

Figures (12)

• Figure 1: RAMSD consists of paired pixel-wise registered SWIR (Short-Wave Infrared) and RGB images captured under various weather conditions (a). Dual modality provides an advantage in different weather and situations (b). We provide benchmarks of our dataset for object detection and domain translation tasks (c).
• Figure 2: Examples from the RASMD dataset: Each pair shows RGB and SWIR views of the same scene. The SWIR camera demonstrates advantages in challenging conditions, making crucial traffic-related objects visible, which are otherwise difficult or impossible to discern in the RGB images.
• Figure 3: Driving Scene "Urban"
• Figure 4: Driving Scene "Suburban"
• Figure 5: Driving Scene "Sunny"