Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Establishing dermatopathology encyclopedia DermpathNet with Artificial Intelligence-Based Workflow

Ziyang Xu, Mingquan Lin, Yiliang Zhou, Zihan Xu, Seth J. Orlow, Zihan Xu, Shane A. Meehan, Alexandra Flamm, Ata S. Moshiri, Yifan Peng

TL;DR

DermpathNet addresses the scarcity of open-access dermatopathology image datasets by constructing a large, peer-reviewed collection from the PubMed Central Open Access subset using a hybrid AI workflow. The pipeline combines a DenseNet-121 image-modality classifier with keyword-driven caption filtering and culminates in expert curation, yielding 7,772 images across 166 diagnoses with structured metadata. Quantitative evaluation shows the DL classifier (F1 ≈ 0.896) outperforms keyword-based retrieval (F1 ≈ 0.610), while the hybrid approach boosts performance for less common diagnoses (F1 ≈ 0.938). Exploratory GPT-4v testing reveals current multimodal models struggle with dermatopathology, underscoring DermpathNet’s value as an educational resource and a benchmark for advancing pathology-aware AI systems. The dataset, its ongoing updates, and code are publicly available to support learning, cross-referencing, and machine-learning research in dermatopathology.

Abstract

Accessing high-quality, open-access dermatopathology image datasets for learning and cross-referencing is a common challenge for clinicians and dermatopathology trainees. To establish a comprehensive open-access dermatopathology dataset for educational, cross-referencing, and machine-learning purposes, we employed a hybrid workflow to curate and categorize images from the PubMed Central (PMC) repository. We used specific keywords to extract relevant images, and classified them using a novel hybrid method that combined deep learning-based image modality classification with figure caption analyses. Validation on 651 manually annotated images demonstrated the robustness of our workflow, with an F-score of 89.6\% for the deep learning approach, 61.0\% for the keyword-based retrieval method, and 90.4\% for the hybrid approach. We retrieved over 7,772 images across 166 diagnoses and released this fully annotated dataset, reviewed by board-certified dermatopathologists. Using our dataset as a challenging task, we found the current image analysis algorithm from OpenAI inadequate for analyzing dermatopathology images. In conclusion, we have developed a large, peer-reviewed, open-access dermatopathology image dataset, DermpathNet, which features a semi-automated curation workflow.

Establishing dermatopathology encyclopedia DermpathNet with Artificial Intelligence-Based Workflow

TL;DR

DermpathNet addresses the scarcity of open-access dermatopathology image datasets by constructing a large, peer-reviewed collection from the PubMed Central Open Access subset using a hybrid AI workflow. The pipeline combines a DenseNet-121 image-modality classifier with keyword-driven caption filtering and culminates in expert curation, yielding 7,772 images across 166 diagnoses with structured metadata. Quantitative evaluation shows the DL classifier (F1 ≈ 0.896) outperforms keyword-based retrieval (F1 ≈ 0.610), while the hybrid approach boosts performance for less common diagnoses (F1 ≈ 0.938). Exploratory GPT-4v testing reveals current multimodal models struggle with dermatopathology, underscoring DermpathNet’s value as an educational resource and a benchmark for advancing pathology-aware AI systems. The dataset, its ongoing updates, and code are publicly available to support learning, cross-referencing, and machine-learning research in dermatopathology.

Abstract

Accessing high-quality, open-access dermatopathology image datasets for learning and cross-referencing is a common challenge for clinicians and dermatopathology trainees. To establish a comprehensive open-access dermatopathology dataset for educational, cross-referencing, and machine-learning purposes, we employed a hybrid workflow to curate and categorize images from the PubMed Central (PMC) repository. We used specific keywords to extract relevant images, and classified them using a novel hybrid method that combined deep learning-based image modality classification with figure caption analyses. Validation on 651 manually annotated images demonstrated the robustness of our workflow, with an F-score of 89.6\% for the deep learning approach, 61.0\% for the keyword-based retrieval method, and 90.4\% for the hybrid approach. We retrieved over 7,772 images across 166 diagnoses and released this fully annotated dataset, reviewed by board-certified dermatopathologists. Using our dataset as a challenging task, we found the current image analysis algorithm from OpenAI inadequate for analyzing dermatopathology images. In conclusion, we have developed a large, peer-reviewed, open-access dermatopathology image dataset, DermpathNet, which features a semi-automated curation workflow.
Paper Structure (7 sections, 6 figures, 4 tables)

This paper contains 7 sections, 6 figures, 4 tables.

Table of Contents

  1. Development of image modality classification (DL classifier).
  2. Development of a keyword-based classifier.
  3. Development of a hybrid classifier.
  4. Quality control of a human-labeled dermopathology gold standard.
  5. Performance assessment using the human-labeled dermopathology gold standard.
  6. Task 1: True/False questions.
  7. Task 2: Open-ended and multiple-choice questions.

Figures (6)

  • Figure 1: Overview of the Artificial Intelligence empowered workflow for semi-automated retrieval of dermatopathology images from PubMed Central (PMC).
  • Figure 2: Glossary of benign and malignant cutaneous neoplasms of interest for which corresponding articles/images were retrieved and hosted in DermpathNet.
  • Figure 3: Number of articles and images downloaded from PMC-OA pre-filter and published in DermpathNet amongst different categories of cutaneous neoplasms.
  • Figure 4: Comparison of precision, recall, and F-scores across 6 categories of cutaneous neoplasms.
  • Figure 5: Prompts used to assess the image analysis module in GPT-4v in answering True/False, Open-ended, or Multiple-choice questions using randomly selected images from DermpathNet.
  • ...and 1 more figures