IMA++: ISIC Archive Multi-Annotator Dermoscopic Skin Lesion Segmentation Dataset

TL;DR

IMA++ addresses the lack of large-scale multi-annotator segmentation data for dermoscopic skin lesions by releasing the ISIC MultiAnnot++ dataset, comprising 14,967 images and 17,684 masks from 16 annotators, plus consensus masks via MV and STAPLE. It provides rich metadata on annotator identity, tool, and skill level, and offers standardized train/val/test splits stratified by segmentation count and inter-annotator agreement. The work analyzes annotator overlap and agreement patterns, compares IMA++ to other public datasets, and demonstrates its potential for annotator-specific modeling, consensus learning, and multi-modal analyses. By making data and code available, IMA++ enables robust exploration of multi-annotator segmentation, annotation style discovery, and the impact of segmentation variability on clinical metrics and melanoma detection.

Abstract

Multi-annotator medical image segmentation is an important research problem, but requires annotated datasets that are expensive to collect. Dermoscopic skin lesion imaging allows human experts and AI systems to observe morphological structures otherwise not discernable from regular clinical photographs. However, currently there are no large-scale publicly available multi-annotator skin lesion segmentation (SLS) datasets with annotator-labels for dermoscopic skin lesion imaging. We introduce ISIC MultiAnnot++, a large public multi-annotator skin lesion segmentation dataset for images from the ISIC Archive. The final dataset contains 17,684 segmentation masks spanning 14,967 dermoscopic images, where 2,394 dermoscopic images have 2-5 segmentations per image, making it the largest publicly available SLS dataset. Further, metadata about the segmentation, including the annotators' skill level and segmentation tool, is included, enabling research on topics such as annotator-specific preference modeling for segmentation and annotator metadata analysis. We provide an analysis on the characteristics of this dataset, curated data partitions, and consensus segmentation masks.

Figures (7)

• Figure 1: A breakdown of the IMA++ dataset: (a) distribution of number of segmentations per image and annotation factor-wise segmentation counts: (b) annotator, (c) tool, and (d) skill level.
• Figure 2: Sample image-segmentation pairs from IMA++: 2 rows each for images with {5, 4, 3, 2} segmentations per image along with the corresponding consensus segmentation masks computed using majority voting (MV) and STAPLE (ST).
• Figure 3: UpSet plot showing the distribution of segmentations across the 16 annotators ("A00" -- "A15"). The distribution is long-tailed, with the top 6 annotators ($\sim$37% of the annotators) contributing $\sim$91% of the segmentations. Best viewed online.
• Figure 4: Quantifying the inter-annotator agreement for IMA++ based on the three factors: (a) annotator, (b) tool, and (c) skill level. For each factor, we report the mean Dice coefficient (left) and 95th percentile of Hausdorff distance (right). Combinations that do not exist in the dataset are grayed out. Best viewed online.
• Figure 5: UpSet plot comparing the proposed IMA++ with eight popular public skin lesion image (both dermoscopic and clinical) segmentation datasets and images shared amongst them. With 14,967, IMA++ is the largest dataset, and although it shares some images with other ISIC Challenge Datasets, over 74% of its images (11,081) are unique from past ISIC challenges. Please also see Table \ref{['tab:datasets-comparison']} for more details about these datasets. Best viewed online.