Efficient and Accurate Hyperspectral Image Demosaicing with Neural Network Architectures

TL;DR

This work tackles the challenge of demosaicing hyperspectral data from MSFAs under real-time constraints. It evaluates three neural network families (lean ResNet, compact U-Net, and parallel architectures) using two MSFA-like datasets, SimpleData and SimRealData, and compares them against classical interpolation and reference CNN methods. The results show that carefully designed, parameter-efficient networks can match or exceed state-of-the-art performance while using substantially fewer parameters, enabling faster, potentially real-time processing. The study also demonstrates that training on MSFA-simulated data—closely aligned with real MSFA characteristics—yields robust spectral reconstruction across datasets, highlighting practical implications for medical imaging and other domains that require accurate spectral information with limited resources.

Abstract

Neural network architectures for image demosaicing have been become more and more complex. This results in long training periods of such deep networks and the size of the networks is huge. These two factors prevent practical implementation and usage of the networks in real-time platforms, which generally only have limited resources. This study investigates the effectiveness of neural network architectures in hyperspectral image demosaicing. We introduce a range of network models and modifications, and compare them with classical interpolation methods and existing reference network approaches. The aim is to identify robust and efficient performing network architectures. Our evaluation is conducted on two datasets, "SimpleData" and "SimRealData," representing different degrees of realism in multispectral filter array (MSFA) data. The results indicate that our networks outperform or match reference models in both datasets demonstrating exceptional performance. Notably, our approach focuses on achieving correct spectral reconstruction rather than just visual appeal, and this emphasis is supported by quantitative and qualitative assessments. Furthermore, our findings suggest that efficient demosaicing solutions, which require fewer parameters, are essential for practical applications. This research contributes valuable insights into hyperspectral imaging and its potential applications in various fields, including medical imaging.

Figures (7)

• Figure 1: Description of the processing pipeline and demosaicing of multispectral cameras.
• Figure 2: The ResNet-based architectures. Top: ResNet according to shinoda. Bottom: our modified version.
• Figure 3: The U-Net-based architecture. A small U-Net structure is used, while instead of the skip connection, we insert the results from ID interpolation in the network.
• Figure 4: The parallel CNN architectures. Top: network Parallel-L is an extension of wisotzky2022hyperspectral using the small ResNet-structure (cf. Fig. \ref{['fig:ResNet']}). Bottom: network Parallel-S is a combination of two effective approaches.
• Figure 5: Visual results and error images. Spectral data are represented in RGB and error images of two region of interest (ROI) are build using $l_1$-Norm. The maximum errors in the top ROI are 0.0598, 0.0994, 0.1001, 0.0805, 0.0667 and in the bottom ROI are 0.0401, 0.0597, 0.0614, 0.0520, 0.0366 in order of appearance of the models from left to right.