Let it shine: Autofluorescence of Papanicolaou-stain improves AI-based cytological oral cancer detection

TL;DR

This study aims to improve AI-based oral cancer detection by introducing additional information through multimodal imaging and deep multimodal information fusion, and demonstrates Intermediate fusion emerges as the leading method among the studied approaches.

Abstract

Oral cancer is a global health challenge. It is treatable if detected early, but it is often fatal in late stages. There is a shift from the invasive and time-consuming tissue sampling and histological examination, toward non-invasive brush biopsies and cytological examination. Reliable computer-assisted methods are essential for cost-effective and accurate cytological analysis, but the lack of detailed cell-level annotations impairs model effectiveness. This study aims to improve AI-based oral cancer detection using multimodal imaging and deep fusion. We combine brightfield and fluorescence whole slide microscopy imaging to analyze Papanicolaou-stained liquid-based cytology slides of brush biopsies collected from both healthy and cancer patients. Due to limited cytological annotations, we utilize a weakly supervised deep learning approach using only patient-level labels. We evaluate various multimodal fusion strategies, including early, late, and three recent intermediate fusion methods. Our results show: (i) fluorescence imaging of Papanicolaou-stained samples provides substantial diagnostic information; (ii) multimodal fusion enhances classification and cancer detection accuracy over single-modality methods. Intermediate fusion is the leading method among the studied approaches. Specifically, the Co-Attention Fusion Network (CAFNet) model excels with an F1 score of 83.34% and accuracy of 91.79%, surpassing human performance on the task. Additional tests highlight the need for precise image registration to optimize multimodal analysis benefits. This study advances cytopathology by combining deep learning and multimodal imaging to enhance early, non-invasive detection of oral cancer, improving diagnostic accuracy and streamlining clinical workflows. The developed pipeline is also applicable in other cytological settings. Our codes and dataset are available online for further research.

Figures (23)

• Figure 1: Illustration of the aligned patch extraction procedure, starting from a rigidly aligned pair of BF and FL WSIs. 1. Nucleus detection in BF (orange lozenges; 31,336 detected in this slide); 2. Patch extraction $256\!\times\!256\,$px in BF (green square); 3. Rigid mapping of nucleus position into FL image; 4. Large $768\!\times\!768\,$px patch extraction (larger blue square); 5. Translation only CMIF-based registration for pixel perfect $256\!\times\!256\,$px patch extraction in FL image; 6. Selection of best the focus level in BF (11 levels) and FL (5 levels). Only three out of the four FL channels (emission wavelengths 668,517,465) are visualized (as RGB).
• Figure 1: Summary of the OC dataset: number of patches and patients, for each class and each partition.
• Figure 2: Classification of microscopy images based on (a) and (b) single-modality; (c) early fusion; (d) late fusion frameworks.
• Figure 2: Average F1 scoreScore, Accuracy, ROC AUC, Recall, and Precision under 3-fold cross-validation on the OC datasetDataset, comparing data augmentation with and without color jitter for the monomodal BF and FL images-only methods.
• Figure 3: Overview of the Multimodal Transfer Module (MMTM) framework,; adapted from joze2020mmtm.