Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)

TL;DR

This paper presents a public benchmark for automated melanoma diagnosis from dermoscopic images, detailing a multi-task challenge (segmentation, dermoscopic feature detection/segmentation, and disease classification) built on ISIC Archive data and hosted on Covalic. It reports 79 submissions from 38 participants across five task parts, with standardized evaluation metrics such as Dice, Jaccard, AUC, and average precision to enable reproducible comparisons. The study demonstrates the potential of automated methods to support lesion annotation, feature detection, and malignant classification, while highlighting the need for further benchmarking against expert variability and blinded clinical assessments. The datasets remain publicly accessible to foster ongoing research and development in melanoma screening and diagnosis.

Abstract

In this article, we describe the design and implementation of a publicly accessible dermatology image analysis benchmark challenge. The goal of the challenge is to sup- port research and development of algorithms for automated diagnosis of melanoma, a lethal form of skin cancer, from dermoscopic images. The challenge was divided into sub-challenges for each task involved in image analysis, including lesion segmentation, dermoscopic feature detection within a lesion, and classification of melanoma. Training data included 900 images. A separate test dataset of 379 images was provided to measure resultant performance of systems developed with the training data. Ground truth for both training and test sets was generated by a panel of dermoscopic experts. In total, there were 79 submissions from a group of 38 participants, making this the largest standardized and comparative study for melanoma diagnosis in dermoscopic images to date. While the official challenge duration and ranking of participants has concluded, the datasets remain available for further research and development.

Figures (4)

• Figure 1: Example dermoscopic images of skin lesions. Typical fields-of-view range from 15-30mm at 10X magnification.
• Figure 2: Example lesion segmentation. Left: original dermoscopic image. Right: binary segmentation mask.
• Figure 3: Example lesion dermoscopic pattern annotations. Left column: original images. Center column: extracted SLIC superpixels. Right column: Positive superpixel annotations highlighted, overlayed over original image. Multiple colors correspond to multiple human annotators. Top row: example for "Globule" annotation label. Bottom row: example for "Streak" annotation label.
• Figure 4: Example lesion classification task. Left: 4 example dermoscopic images of melanoma. Right: 4 example dermoscopic images of benign nevi.