Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantifying uncertainty in lung cancer segmentation with foundation models applied to mixed-domain datasets

Aneesh Rangnekar, Nishant Nadkarni, Jue Jiang, Harini Veeraraghavan

TL;DR

This work addresses the robustness of medical image foundation models for lung tumor segmentation under distribution and concept shifts. It evaluates four SSL-pretrained 3D transformer models (Swin UNETR, SimMIM, iBOT, SMIT) on mixed-domain CT data using a shared fine-tuning setup, comparing ID and OOD performance. Beyond standard segmentation accuracy, it introduces and analyzes OOD metrics such as $AUROC$, $FPR@95$, voxel-wise entropy, and volume occupancy, revealing distinct uncertainty profiles across models. The findings show that incorporating uncertainty and OOD diagnostics provides deeper insight into model behavior, guiding safer clinical deployment when ground-truth labels are not readily available.

Abstract

Medical image foundation models have shown the ability to segment organs and tumors with minimal fine-tuning. These models are typically evaluated on task-specific in-distribution (ID) datasets. However, reliable performance on ID datasets does not guarantee robust generalization on out-of-distribution (OOD) datasets. Importantly, once deployed for clinical use, it is impractical to have `ground truth' delineations to assess ongoing performance drifts, especially when images fall into the OOD category due to different imaging protocols. Hence, we introduced a comprehensive set of computationally fast metrics to evaluate the performance of multiple foundation models (Swin UNETR, SimMIM, iBOT, SMIT) trained with self-supervised learning (SSL). All models were fine-tuned on identical datasets for lung tumor segmentation from computed tomography (CT) scans. The evaluation was performed on two public lung cancer datasets (LRAD: n = 140, 5Rater: n = 21) with different image acquisitions and tumor stages compared to training data (n = 317 public resource with stage III-IV lung cancers) and a public non-cancer dataset containing volumetric CT scans of patients with pulmonary embolism (n = 120). All models produced similarly accurate tumor segmentation on the lung cancer testing datasets. SMIT produced the highest F1-score (LRAD: 0.60, 5Rater: 0.64) and lowest entropy (LRAD: 0.06, 5Rater: 0.12), indicating higher tumor detection rate and confident segmentations. In the OOD dataset, SMIT misdetected the least number of tumors, marked by a median volume occupancy of 5.67 cc compared to the best method SimMIM of 9.97 cc. Our analysis shows that additional metrics such as entropy and volume occupancy may help better understand model performance on mixed domain datasets.

Quantifying uncertainty in lung cancer segmentation with foundation models applied to mixed-domain datasets

TL;DR

This work addresses the robustness of medical image foundation models for lung tumor segmentation under distribution and concept shifts. It evaluates four SSL-pretrained 3D transformer models (Swin UNETR, SimMIM, iBOT, SMIT) on mixed-domain CT data using a shared fine-tuning setup, comparing ID and OOD performance. Beyond standard segmentation accuracy, it introduces and analyzes OOD metrics such as , , voxel-wise entropy, and volume occupancy, revealing distinct uncertainty profiles across models. The findings show that incorporating uncertainty and OOD diagnostics provides deeper insight into model behavior, guiding safer clinical deployment when ground-truth labels are not readily available.

Abstract

Medical image foundation models have shown the ability to segment organs and tumors with minimal fine-tuning. These models are typically evaluated on task-specific in-distribution (ID) datasets. However, reliable performance on ID datasets does not guarantee robust generalization on out-of-distribution (OOD) datasets. Importantly, once deployed for clinical use, it is impractical to have `ground truth' delineations to assess ongoing performance drifts, especially when images fall into the OOD category due to different imaging protocols. Hence, we introduced a comprehensive set of computationally fast metrics to evaluate the performance of multiple foundation models (Swin UNETR, SimMIM, iBOT, SMIT) trained with self-supervised learning (SSL). All models were fine-tuned on identical datasets for lung tumor segmentation from computed tomography (CT) scans. The evaluation was performed on two public lung cancer datasets (LRAD: n = 140, 5Rater: n = 21) with different image acquisitions and tumor stages compared to training data (n = 317 public resource with stage III-IV lung cancers) and a public non-cancer dataset containing volumetric CT scans of patients with pulmonary embolism (n = 120). All models produced similarly accurate tumor segmentation on the lung cancer testing datasets. SMIT produced the highest F1-score (LRAD: 0.60, 5Rater: 0.64) and lowest entropy (LRAD: 0.06, 5Rater: 0.12), indicating higher tumor detection rate and confident segmentations. In the OOD dataset, SMIT misdetected the least number of tumors, marked by a median volume occupancy of 5.67 cc compared to the best method SimMIM of 9.97 cc. Our analysis shows that additional metrics such as entropy and volume occupancy may help better understand model performance on mixed domain datasets.
Paper Structure (4 sections, 3 figures, 2 tables)

This paper contains 4 sections, 3 figures, 2 tables.

Table of Contents

  1. Purpose/Motivation
  2. Methods
  3. Results
  4. Conclusion

Figures (3)

  • Figure 1: Example segmentations and voxel-wise entropy shown for superior, central, and inferior slices. The average entropy for individual slices is shown on top-right. Volumetric DSC for the entire image is shown in yellow on the bottom-left. The '-' indicates that the model did not detect/segment any tumor.
  • Figure 2: Example segmentations and corresponding entropy for the four models on two different images from the Pulmonary Embolism dataset. The slice-wise entropy is displayed in orange on the top-right. The '-' indicates that the model did not detect/segment any tumor.
  • Figure 3: UMAP clustering for features of the validation and Pulmonary embolism images derived from Stage 1 of the Swin transformer architecture for all four foundation models.