CleanPatrick: A Benchmark for Image Data Cleaning

TL;DR

CleanPatrick addresses the lack of realistic image data-cleaning benchmarks by constructing a large-scale, ground-truth annotated benchmark from Fitzpatrick17k. It jointly annotates off-topic samples, near duplicates, and label errors via medical crowd workers and expert validation, using an item-response theory–inspired aggregation and expert-calibrated thresholds. By formalizing each detection task as a ranking problem and evaluating across metrics like AUROC and AP, the study reveals that near-duplicate detection benefits from self-supervised representations, off-topic detection aligns with classical anomaly detectors, and label-error detection remains a challenging open problem in medical imaging. The released dataset and evaluation framework enable fair, cross-method comparisons and pave the way for more reliable data-centric AI in the medical imaging domain.

Abstract

Robust machine learning depends on clean data, yet current image data cleaning benchmarks rely on synthetic noise or narrow human studies, limiting comparison and real-world relevance. We introduce CleanPatrick, the first large-scale benchmark for data cleaning in the image domain, built upon the publicly available Fitzpatrick17k dermatology dataset. We collect 496,377 binary annotations from 933 medical crowd workers, identify off-topic samples (4%), near-duplicates (21%), and label errors (22%), and employ an aggregation model inspired by item-response theory followed by expert review to derive high-quality ground truth. CleanPatrick formalizes issue detection as a ranking task and adopts typical ranking metrics mirroring real audit workflows. Benchmarking classical anomaly detectors, perceptual hashing, SSIM, Confident Learning, NoiseRank, and SelfClean, we find that, on CleanPatrick, self-supervised representations excel at near-duplicate detection, classical methods achieve competitive off-topic detection under constrained review budgets, and label-error detection remains an open challenge for fine-grained medical classification. By releasing both the dataset and the evaluation framework, CleanPatrick enables a systematic comparison of image-cleaning strategies and paves the way for more reliable data-centric artificial intelligence.

Figures (8)

• Figure 1: Process of acquiring and curating the CleanPatrick benchmark. We started by collecting annotations from medical crowd workers for three types of data quality issues. This was followed by a probabilistic estimation of sample quality and annotator expertise, used to aggregate the collected annotations. Finally, a group of medical domain experts judged the quality of a subsample of the dataset.
• Figure 2: Examples of data quality issues identified in the Fitzpatrick17k dataset.Off-topic samples include unrelated content (e.g., laboratory equipment, diagrams) and images with insufficient diagnostic value. Near duplicates comprise identical images at different resolutions (thumbnails) and multiple photographs of the same condition from different angles. Label errors show both clear mislabelings and rare conditions that were incorrectly classified or assigned. These naturally occurring issues form the foundation of the CleanPatrick benchmark, providing a realistic test scenario for evaluating data cleaning algorithms across varying levels of detection difficulty.
• Figure 3: Performance of different data cleaning approaches (represented in colors) for the three quality issues investigated for different ranking metrics (P@100, P@1000, AUROC, and AP). Methods are separated into data quality issue-specific ones and holistic methods able to detect multiple issues. The dotted lines refer to the uninformed baseline, which randomly shuffles the ranking.
• Figure 4: Left shows a screenshot of the labeling interface shown to the medical crowd workers. Right shows the instructions given to the annotators for the respective labeling tasks for the data quality issues. Along with each set of instructions, the annotators were given some example images of both the positive and negative responses.
• Figure 5: Histograms showing the number of annotations from medical crowd workers per image sample for each data quality issue.

Theorems & Definitions (1)

• Lemma 1