Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Challenges in Deep Learning-Based Small Organ Segmentation: A Benchmarking Perspective for Medical Research with Limited Datasets

Phongsakon Mark Konrad, Andrei-Alexandru Popa, Yaser Sabzehmeidani, Liang Zhong, Madhulika Tripathy, Andrei Constantinescu, Elisa A. Liehn, Serkan Ayvaz

Abstract

Accurate segmentation of carotid artery structures in histopathological images is vital for cardiovascular disease research. This study systematically evaluates ten deep learning segmentation models including classical architectures, modern CNNs, a Vision Transformer, and foundation models, on a limited dataset of nine cardiovascular histology images. We conducted ablation studies on data augmentation, input resolution, and random seed stability to quantify sources of variance. Evaluation on an independent generalization dataset ($N=153$) under distribution shift reveals that foundation models maintain performance while classical architectures fail, and that rankings change substantially between in-distribution and out-of-distribution settings. Training on the second dataset at varying sample sizes reveals dataset-specific ranking hierarchies confirming that model rankings are not generalizable across datasets. Despite rigorous Bayesian hyperparameter optimization, model performance remains highly sensitive to data splits. The bootstrap analysis reveals substantially overlapping confidence intervals among top models, with differences driven more by statistical noise than algorithmic superiority. This instability exposes limitations of standard benchmarking in low-data clinical settings and challenges assumptions that performance rankings reflect clinical utility. We advocate for uncertainty-aware evaluation in low-data clinical research scenarios from two perspectives. First, the scenario is not niche and is rather widely spread; and second, it enables pursuing or discontinuing research tracks with limited datasets from incipient stages of observations.

Challenges in Deep Learning-Based Small Organ Segmentation: A Benchmarking Perspective for Medical Research with Limited Datasets

Abstract

Accurate segmentation of carotid artery structures in histopathological images is vital for cardiovascular disease research. This study systematically evaluates ten deep learning segmentation models including classical architectures, modern CNNs, a Vision Transformer, and foundation models, on a limited dataset of nine cardiovascular histology images. We conducted ablation studies on data augmentation, input resolution, and random seed stability to quantify sources of variance. Evaluation on an independent generalization dataset () under distribution shift reveals that foundation models maintain performance while classical architectures fail, and that rankings change substantially between in-distribution and out-of-distribution settings. Training on the second dataset at varying sample sizes reveals dataset-specific ranking hierarchies confirming that model rankings are not generalizable across datasets. Despite rigorous Bayesian hyperparameter optimization, model performance remains highly sensitive to data splits. The bootstrap analysis reveals substantially overlapping confidence intervals among top models, with differences driven more by statistical noise than algorithmic superiority. This instability exposes limitations of standard benchmarking in low-data clinical settings and challenges assumptions that performance rankings reflect clinical utility. We advocate for uncertainty-aware evaluation in low-data clinical research scenarios from two perspectives. First, the scenario is not niche and is rather widely spread; and second, it enables pursuing or discontinuing research tracks with limited datasets from incipient stages of observations.

Paper Structure

This paper contains 62 sections, 15 equations, 24 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. CNN-based Architectures for Medical Segmentation
  4. Transformer-based Architectures in Vision and Medicine
  5. Foundation Models and Their Adaptation for Medical Imaging
  6. Hyperparameter Optimization in Medical AI
  7. The Benchmarking Reliability on Small Medical Datasets
  8. A Systematic Model Evaluation Framework
  9. Dataset and Clinical Context
  10. Data Preprocessing and Augmentation
  11. Model Architecture Selection
  12. Classical CNN Architectures
  13. Modern CNN Architectures
  14. Modern Vision Transformer
  15. Foundation Models and Hybrid Architectures
  16. ...and 47 more sections

Figures (24)

  • Figure 1: The systematic evaluation framework. Our process begins with (1) data collection and expert annotation. After the data preparation (2.1) Each model architecture then undergoes (2.2) extensive Bayesian hyperparameter optimization to find its optimal configuration and then use it for the training (2.3). (3.1) We then evaluate the segmentation performances, which are then dissected using our (3.2.) multi-modal XAI framework for qualitative insight.
  • Figure 2: Example of data preparation. (A) Original histological image. (B) Expert line-art annotation. (C) Processed ground truth segmentation mask for Lumen (yellow), Neointima (red), and Media (blue).
  • Figure 3: 5 µm serial sections were collected starting from the bifurcation from the atherosclerotic injured common carotid artery, 50 µm apart, as represented. Created in BioRender. Liehn, E. (2026) https://BioRender.com/sm3xslo.
  • Figure 4: The impact of hyperparameter selection. All twelve segmentation results were generated by the same MedSAM architecture. The wide variation in output quality is due solely to different hyperparameter configurations. This illustrates that without a rigorous search, one could incorrectly dismiss a capable model based on a single, suboptimal trial.
  • Figure 5: The five-layer XAI framework. (A) A model's prediction is dissected into (1) Error Analysis, (2) Uncertainty, (3) Morphology, (4) Attention, and (5) Saliency. (B) These are synthesized into an integrated explanation, providing a multi-faceted view of the model's behavior.
  • ...and 19 more figures