Swin transformers are robust to distribution and concept drift in endoscopy-based longitudinal rectal cancer assessment

TL;DR

The paper tackles variability in endoscopy-based rectal cancer assessment across treatment and follow-up by training a pretrained Swin transformer to classify tumor vs non-tumor in longitudinal images. It introduces an optimal transport–based color-shift test to simulate distribution drift and evaluates robustness to concept drift using follow-up and external datasets, comparing Swin against ResNet-50, Wide ResNet-50, and ViT. The Swin model demonstrates robust performance under color and concept shifts, outperforming other architectures on both in-distribution and out-of-distribution data, indicating potential for reliable longitudinal cancer monitoring from endoscopy. Together, these findings support Swin transformers as a practical tool for objective, drift-robust rectal cancer assessment in endoscopic imaging.

Abstract

Endoscopic images are used at various stages of rectal cancer treatment starting from cancer screening, diagnosis, during treatment to assess response and toxicity from treatments such as colitis, and at follow up to detect new tumor or local regrowth (LR). However, subjective assessment is highly variable and can underestimate the degree of response in some patients, subjecting them to unnecessary surgery, or overestimate response that places patients at risk of disease spread. Advances in deep learning has shown the ability to produce consistent and objective response assessment for endoscopic images. However, methods for detecting cancers, regrowth, and monitoring response during the entire course of patient treatment and follow-up are lacking. This is because, automated diagnosis and rectal cancer response assessment requires methods that are robust to inherent imaging illumination variations and confounding conditions (blood, scope, blurring) present in endoscopy images as well as changes to the normal lumen and tumor during treatment. Hence, a hierarchical shifted window (Swin) transformer was trained to distinguish rectal cancer from normal lumen using endoscopy images. Swin as well as two convolutional (ResNet-50, WideResNet-50), and vision transformer (ViT) models were trained and evaluated on follow-up longitudinal images to detect LR on private dataset as well as on out-of-distribution (OOD) public colonoscopy datasets to detect pre/non-cancerous polyps. Color shifts were applied using optimal transport to simulate distribution shifts. Swin and ResNet models were similarly accurate in the in-distribution dataset. Swin was more accurate than other methods (follow-up: 0.84, OOD: 0.83) even when subject to color shifts (follow-up: 0.83, OOD: 0.87), indicating capability to provide robust performance for longitudinal cancer assessment.

Figures (2)

• Figure 1: Confusion matrix for classifying ID testing dataset by the analyzed DL models for images subjected to (a) no color shift and (b) with color shift. TN: true negatives, FP: false positives, FN: false negatives, and TP: true positives.
• Figure 2: Effect of color shifting on ResNet-50 and Swin predictions in both tumor and no-tumor images at different stages of treatment and the external dataset.