Leveraging Computational Pathology AI for Noninvasive Optical Imaging Analysis Without Retraining

TL;DR

This paper introduces FoundationShift, a method to apply any AI model from computational pathology without retraining to noninvasive in vivo images, and shows it is more accurate than state of the art models with multiple imaging modalities, with multiple imaging modalities.

Abstract

Noninvasive optical imaging modalities can probe patient's tissue in 3D and over time generate gigabytes of clinically relevant data per sample. There is a need for AI models to analyze this data and assist clinical workflow. The lack of expert labelers and the large dataset required (>100,000 images) for model training and tuning are the main hurdles in creating foundation models. In this paper we introduce FoundationShift, a method to apply any AI model from computational pathology without retraining. We show our method is more accurate than state of the art models (SAM, MedSAM, SAM-Med2D, CellProfiler, Hover-Net, PLIP, UNI and ChatGPT), with multiple imaging modalities (OCT and RCM). This is achieved without the need for model retraining or fine-tuning. Applying our method to noninvasive in vivo images could enable physicians to readily incorporate optical imaging modalities into their clinical practice, providing real time tissue analysis and improving patient care.

Figures (13)

• Figure 1: Tissue segmentation of OCT images utilizing FoundationShift and off-the-shelf segmentation models.a) Off-the-shelf segmentation foundation models perform poorly when segmenting OCT images. b) By performing domain transfer first, we are able to significantly improve segmentation accuracy. c,d) Segmentation examples of off-the-shelf algorithms compared to our approach (Domain Transfer + MedSAM). Ground truth expert epidermis segmentation is highlighted in green contour. Algorithm segmentation is visualized in blue. e) Dice similarity coefficient quantification for each algorithm. The center line within the colored box represents the median value, with the bottom and top bounds of the box delineating the 25th and 75th percentiles, respectively, whiskers represent minimum and maximum scores over the 95 sections. Domain transfer followed by MedSAM outperforms all off-the-shelf algorithms tested ($~p<2\cdot10^{-15}$). f) Domain transfer + MedSAM perform 3D segmentation of skin sample containing a Basal Cell Carcinoma nodule (orange). Epidermis segmentation visualized in blue. g,h) 2D cross-sectional views from within the volume, which can be chosen to slice across any plane. Scale bars in c,f,h) are 200µm.
• Figure 2: Cell segmentation of RCM images utilizing FoundationShift and off-the-shelf model.a) An RCM image obtained by Li et al. Cells are very difficult to visualize. Off-the-shelf cell segmentation Hover-Net is unable to segment any cells. b) By performing domain transfer first, we are able to significantly improve segmentation accuracy. We are able to reliably segment cells without any model tuning (zero shot). c) Comparing cell segmentation accuracy of three models: Hover-Net evaluated on RCM images, CellProfiler, and our method. Our method outperforms other methods in all quality parameters: Dice score, detection quality (DQ), segmentation quality (SQ), and panoptic quality (PQ). d) Domain transfer + Hover-Net perform 3D segmentation of cells in Epidermis. Scale bar in a) is 10µm. Scale bar in d) is 50µm.
• Figure : Visual Abstract
• Figure S1: OCT2Hist: domain transfer from OCT to virtual H&E. We show examples of an OCT image (left column), the corresponding domain transfer computer generated virtual H&E image (middle column), and the corresponding ground truth histology image (right column) from a few skin samples. Dermal epidermal junction is visible in both OCT and ground truth H&E and is reproduced by virtual H&E (arrows). Scale bar: 200µm.
• Figure S2: Measuring the Kullback Leibler (KL) divergence between OCT and histology.a) The KL divergence between OCT images and H&E images from the same locations, before and after domain transfer. The higher KL divergence between OCT and H&E before domain transfer indicates a greater dissimilarity, while the reduced divergence after domain transfer suggests a closer alignment to the H&E training domain. b) A comparison of KL divergence in a related study performing domain transfer from CT to MR jin2019deep, showing less significant improvement compared to the OCT to H&E domain transfer. These results highlight the substantial impact of domain transfer in aligning images closer to the training domain, enhancing the performance of computational pathology models.