Exploring Fully Convolutional Networks for the Segmentation of Hyperspectral Imaging Applied to Advanced Driver Assistance Systems

TL;DR

This work examines fully convolutional networks for segmenting hyperspectral driving scenes acquired by a small snapshot camera, aiming to enhance ADAS under challenging conditions. By adapting a U-Net to the HSI-Drive v1.1 dataset and performing hyperparameter optimization, the authors demonstrate that spectro-spatial learning substantially outperforms purely spectral classifiers, with overlapping patches further boosting accuracy. A complete rapid-prototyping workflow on a Xilinx MPSoC, including PREprocessing, ONNX/Keras deployment, and 8-bit quantization, achieves near real-time performance (up to $487.91$ FPS on hardware, and $27$ FPS when including preprocessing), illustrating practical viability for on-board ADAS. The findings suggest that while spectral information helps, the spatial features learned by FCNs dominate performance, motivating future work on 3D convolutions, multiscale fusion, and optimized data transfer to further close the gap in tricky scenes with shadows or overlapping materials.

Abstract

Advanced Driver Assistance Systems (ADAS) are designed with the main purpose of increasing the safety and comfort of vehicle occupants. Most of current computer vision-based ADAS perform detection and tracking tasks quite successfully under regular conditions, but are not completely reliable, particularly under adverse weather and changing lighting conditions, neither in complex situations with many overlapping objects. In this work we explore the use of hyperspectral imaging (HSI) in ADAS on the assumption that the distinct near infrared (NIR) spectral reflectances of different materials can help to better separate the objects in a driving scene. In particular, this paper describes some experimental results of the application of fully convolutional networks (FCN) to the image segmentation of HSI for ADAS applications. More specifically, our aim is to investigate to what extent the spatial features codified by convolutional filters can be helpful to improve the performance of HSI segmentation systems. With that aim, we use the HSI-Drive v1.1 dataset, which provides a set of labelled images recorded in real driving conditions with a small-size snapshot NIR-HSI camera. Finally, we analyze the implementability of such a HSI segmentation system by prototyping the developed FCN model together with the necessary hyperspectral cube preprocessing stage and characterizing its performance on an MPSoC.

Figures (4)

• Figure 1: Architecture of the modified U-Net.
• Figure 2: Comparison among the visible (first row), 3-class ground truth (second row), 3-class U-Net segmentation (third row), 5-class ground truth (fourth row) and 5-class U-Net segmentation (fifth row) images of three different scenarios: urban (first column), road (second column) and highway (third column).
• Figure 3: Overall accuracy (%) as a function of the number of spectral channels.
• Figure 4: Segmented images produced by the deployed model on the MPSoC.