Multispectral Stereo-Image Fusion for 3D Hyperspectral Scene Reconstruction

TL;DR

A novel approach combining two calibrated multispectral real-time capable snapshot cameras, covering different spectral ranges, into a stereo-system, which enables both 3D reconstruction and spectral analysis and investigates its applicability for surgical assistance monitoring.

Abstract

Spectral imaging enables the analysis of optical material properties that are invisible to the human eye. Different spectral capturing setups, e.g., based on filter-wheel, push-broom, line-scanning, or mosaic cameras, have been introduced in the last years to support a wide range of applications in agriculture, medicine, and industrial surveillance. However, these systems often suffer from different disadvantages, such as lack of real-time capability, limited spectral coverage or low spatial resolution. To address these drawbacks, we present a novel approach combining two calibrated multispectral real-time capable snapshot cameras, covering different spectral ranges, into a stereo-system. Therefore, a hyperspectral data-cube can be continuously captured. The combined use of different multispectral snapshot cameras enables both 3D reconstruction and spectral analysis. Both captured images are demosaicked avoiding spatial resolution loss. We fuse the spectral data from one camera into the other to receive a spatially and spectrally high resolution video stream. Experiments demonstrate the feasibility of this approach and the system is investigated with regard to its applicability for surgical assistance monitoring.

Figures (8)

• Figure 1: The demosaicing network. Two parallel feature extracting layers using a mosaic-to-cube converter (M2C) on one side and ResNet blocks on the other side are used followed by a feature adding and two deconvolution (deconv) layers to upsample the spatial dimensions of the image.
• Figure 2: The entire pipeline to calculate disparity. The first part includes parallel demosaicing of both camera images. Then a feature extraction follows using a mixture of convolutional layers and ResNet units. As last step the disparity is calculated iteratively by using several convolutional blocks.
• Figure 3: The stereo-HSI-setup. Left: the setup mounted on a surgical retractor system in the operation room. Right: detailed view of the stereo-cameras mounted on the rigid trail.
• Figure 4: Calibration results. The top figure shows an example of a difference image between the projected ground truth checkerboard data and a captured left ($4$$\times$$4$ mosaic snapshot) camera view. The bottom figure shows an example of the re-projected corner points of the checkerboard. The detected corner points (blue) fit almost perfectly with the re-projected corner points (red).
• Figure 5: One spectral channel of the right $4$$\times$$4$ camera view after warping into the other camera view. Due to the warping, some areas at the image borders hold no data and are repeated by neighboring pixels. To avoid this effect during analysis, the final data cube is cropped.