On-chip Hyperspectral Image Segmentation with Fully Convolutional Networks for Scene Understanding in Autonomous Driving

TL;DR

This work demonstrates that on‑chip hyperspectral image segmentation for ADAS is feasible with a lightweight FCN when combined with spectral data preprocessing and careful hardware‑aware design. By comparing baseline spectral classifiers to a compact U‑Net variant and evaluating across three embedded platforms, the authors show that incorporating spatial context substantially improves segmentation under real driving conditions, while keeping model size modest. A full prototype pipeline is benchmarked, including preprocessing, quantization, and deployment on Raspberry Pi, Jetson Nano, and an FPGA‑based MPSoC, with the FPGA delivering the best latency/energy profile and achieving around 20 FPS. The study highlights practical constraints in preprocessing and data labeling, and points to future improvements via dataset enrichment and quantization/training strategies to further enhance robustness and real‑time performance.

Abstract

Most of current computer vision-based advanced driver assistance systems (ADAS) perform detection and tracking of objects quite successfully under regular conditions. However, under adverse weather and changing lighting conditions, and in complex situations with many overlapping objects, these systems are not completely reliable. The spectral reflectance of the different objects in a driving scene beyond the visible spectrum can offer additional information to increase the reliability of these systems, especially under challenging driving conditions. Furthermore, this information may be significant enough to develop vision systems that allow for a better understanding and interpretation of the whole driving scene. In this work we explore the use of snapshot, video-rate hyperspectral imaging (HSI) cameras in ADAS on the assumption that the near infrared (NIR) spectral reflectance of different materials can help to better segment the objects in real driving scenarios. To do this, we have used the HSI-Drive 1.1 dataset to perform various experiments on spectral classification algorithms. However, the information retrieval of hyperspectral recordings in natural outdoor scenarios is challenging, mainly because of deficient colour constancy and other inherent shortcomings of current snapshot HSI technology, which poses some limitations to the development of pure spectral classifiers. In consequence, in this work we analyze to what extent the spatial features codified by standard, tiny fully convolutional network (FCN) models can improve the performance of HSI segmentation systems for ADAS applications. The abstract above is truncated due to submission limits. For the full abstract, please refer to the published article.

Figures (18)

• Figure 1: Ground truth image (a) and its corresponding false-color visible image (b) of an urban scenario as an example of the weak labelling methodology followed to develop the HSI-Drive dataset.
• Figure 2: Pearson Correlation Coefficients among the 25 spectral bands (a) and spectral signature of some of the representative classes (b) of the HSI-Drive 1.1 dataset.
• Figure 3: Intraclass Pearson Correlation Coefficients for the 25 bands of two representative classes of the dataset: Road Marks (a) and Painted Metal (b).
• Figure 4: Comparison of t-SNE output when using either the 3 most representative channels (a) or the full 25 channels (b).
• Figure 5: Overall, average and weighted IoU (%) as a function of the number of spectral channels.