Automatic Tissue Differentiation in Parotidectomy using Hyperspectral Imaging

TL;DR

This study addresses intraoperative tissue differentiation during parotidectomy to minimize injury to nerves and vessels. It uses a stereo-HSI system to form a hyperspectral data cube spanning $400$-$1000$ nm with $41$ spectral bands, processed by a six-layer 3D-CNN to classify five tissue types across patches of size $31×31×41$. Training on data from $18$ surgeries (27 images) with leave-one-subject-out evaluation yielded a validation accuracy of $98.7\%$ and an unseen-subject accuracy of $83.4\%$, with skin and nerve tissues showing high sensitivity and veins being more challenging. The work demonstrates the feasibility of real-time spectral tissue differentiation in the operating room and highlights the need for more balanced data and post-processing to handle localized reflections and improve robustness, potentially enabling AR-based intraoperative guidance. $41$ spectral bands across the $400$-$1000$ nm range and spatially resolved holographic overlays could enhance surgical decision-making and safety.

Abstract

In head and neck surgery, continuous intraoperative tissue differentiation is of great importance to avoid injury to sensitive structures such as nerves and vessels. Hyperspectral imaging (HSI) with neural network analysis could support the surgeon in tissue differentiation. A 3D Convolutional Neural Network with hyperspectral data in the range of $400-1000$ nm is used in this work. The acquisition system consisted of two multispectral snapshot cameras creating a stereo-HSI-system. For the analysis, 27 images with annotations of glandular tissue, nerve, muscle, skin and vein in 18 patients undergoing parotidectomy are included. Three patients are removed for evaluation following the leave-one-subject-out principle. The remaining images are used for training, with the data randomly divided into a training group and a validation group. In the validation, an overall accuracy of $98.7\%$ is achieved, indicating robust training. In the evaluation on the excluded patients, an overall accuracy of $83.4\%$ has been achieved showing good detection and identification abilities. The results clearly show that it is possible to achieve robust intraoperative tissue differentiation using hyperspectral imaging. Especially the high sensitivity in parotid or nerve tissue is of clinical importance. It is interesting to note that vein was often confused with muscle. This requires further analysis and shows that a very good and comprehensive data basis is essential. This is a major challenge, especially in surgery.

Figures (5)

• Figure 1: The classical RGB representation of an captured hyperspectral image.
• Figure 2: The different tissue regions, annotated by the surgeon, are bordered using specific color coding.
• Figure 3: The distribution of all recorded tissue patches according to their categorization into the five tissue classes.
• Figure 4: The confusion matrix of the best performing model applied on the evaluation patches showing an overall classification accuracy of $83.4\%$.
• Figure 5: Predicted areas in an evaluation image, cf. Fig. \ref{['img:annotate']}.