Low-cost Robust Night-time Aerial Material Segmentation through Hyperspectral Data and Sparse Spatio-Temporal Learning

TL;DR

This work proposes an innovative Siamese framework that uses time series-based compression to effectively and scalably integrate the additional spectral data into the segmentation task.

Abstract

Material segmentation is a complex task, particularly when dealing with aerial data in poor lighting and atmospheric conditions. To address this, hyperspectral data from specialized cameras can be very useful in addition to RGB images. However, due to hardware constraints, high spectral data often come with lower spatial resolution. Additionally, incorporating such data into a learning-based segmentation framework is challenging due to the numerous data channels involved. To overcome these difficulties, we propose an innovative Siamese framework that uses time series-based compression to effectively and scalably integrate the additional spectral data into the segmentation task. We demonstrate our model's effectiveness through competitive benchmarks on aerial datasets in various environmental conditions.

Figures (6)

• Figure 1: General framework pipeline
• Figure 2: The most informative timestamps and values for the timeseries induced from the original HSI image on each material class
• Figure 3: Original and reconstructed material segmentation map using Motion Code
• Figure 4: Adverse image effects added incrementally from left to right on the Jasper Ridge dataset
• Figure 5: gDice performance comparison of models with varying levels of darkness and atmospheric scattering on Jasper Ridge dataset, higher $A$ leads to higher scattering. Our model (blue) performs consistently. Meanwhile, degradation of RGB only models in low-light scenarios is drastic.