Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MAKE: Multi-Aspect Knowledge-Enhanced Vision-Language Pretraining for Zero-shot Dermatological Assessment

Siyuan Yan, Xieji Li, Ming Hu, Yiwen Jiang, Zhen Yu, Zongyuan Ge

TL;DR

Dermatology demands fusing visual skin features with rich clinical knowledge, but conventional vision-language pretraining is limited by text length and unstructured clinical narratives. MAKE introduces a threefold approach—multi-aspect knowledge-image contrastive learning, fine-grained alignment, and diagnosis-guided weighting—to align diverse clinical captions with image patches in a zero-shot setting, trained on 403,563 dermatology image-text pairs. Across eight datasets, MAKE achieves superior performance on zero-shot skin disease classification, concept annotation, and cross-modal retrieval, with notable gains over baselines and robust ablation-supported contributions. This work advances practical zero-shot dermatology reasoning and provides a scalable framework for integrating multi-attribute medical knowledge into vision-language models.

Abstract

Dermatological diagnosis represents a complex multimodal challenge that requires integrating visual features with specialized clinical knowledge. While vision-language pretraining (VLP) has advanced medical AI, its effectiveness in dermatology is limited by text length constraints and the lack of structured texts. In this paper, we introduce MAKE, a Multi-Aspect Knowledge-Enhanced vision-language pretraining framework for zero-shot dermatological tasks. Recognizing that comprehensive dermatological descriptions require multiple knowledge aspects that exceed standard text constraints, our framework introduces: (1) a multi-aspect contrastive learning strategy that decomposes clinical narratives into knowledge-enhanced sub-texts through large language models, (2) a fine-grained alignment mechanism that connects subcaptions with diagnostically relevant image features, and (3) a diagnosis-guided weighting scheme that adaptively prioritizes different sub-captions based on clinical significance prior. Through pretraining on 403,563 dermatological image-text pairs collected from education resources, MAKE significantly outperforms state-of-the-art VLP models on eight datasets across zero-shot skin disease classification, concept annotation, and cross-modal retrieval tasks. Our code will be made publicly available at https: //github.com/SiyuanYan1/MAKE.

MAKE: Multi-Aspect Knowledge-Enhanced Vision-Language Pretraining for Zero-shot Dermatological Assessment

TL;DR

Dermatology demands fusing visual skin features with rich clinical knowledge, but conventional vision-language pretraining is limited by text length and unstructured clinical narratives. MAKE introduces a threefold approach—multi-aspect knowledge-image contrastive learning, fine-grained alignment, and diagnosis-guided weighting—to align diverse clinical captions with image patches in a zero-shot setting, trained on 403,563 dermatology image-text pairs. Across eight datasets, MAKE achieves superior performance on zero-shot skin disease classification, concept annotation, and cross-modal retrieval, with notable gains over baselines and robust ablation-supported contributions. This work advances practical zero-shot dermatology reasoning and provides a scalable framework for integrating multi-attribute medical knowledge into vision-language models.

Abstract

Dermatological diagnosis represents a complex multimodal challenge that requires integrating visual features with specialized clinical knowledge. While vision-language pretraining (VLP) has advanced medical AI, its effectiveness in dermatology is limited by text length constraints and the lack of structured texts. In this paper, we introduce MAKE, a Multi-Aspect Knowledge-Enhanced vision-language pretraining framework for zero-shot dermatological tasks. Recognizing that comprehensive dermatological descriptions require multiple knowledge aspects that exceed standard text constraints, our framework introduces: (1) a multi-aspect contrastive learning strategy that decomposes clinical narratives into knowledge-enhanced sub-texts through large language models, (2) a fine-grained alignment mechanism that connects subcaptions with diagnostically relevant image features, and (3) a diagnosis-guided weighting scheme that adaptively prioritizes different sub-captions based on clinical significance prior. Through pretraining on 403,563 dermatological image-text pairs collected from education resources, MAKE significantly outperforms state-of-the-art VLP models on eight datasets across zero-shot skin disease classification, concept annotation, and cross-modal retrieval tasks. Our code will be made publicly available at https: //github.com/SiyuanYan1/MAKE.
Paper Structure (9 sections, 8 equations, 2 figures, 3 tables)

This paper contains 9 sections, 8 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Encoding Stage
  4. Multi-aspect Knowledge-Image Contrastive Learning
  5. Fine-grained Alignment
  6. Diagnosis Knowledge-guided Weighting
  7. Experimental details
  8. Results
  9. Conclusion

Figures (2)

  • Figure 1: Comparison between training strategies. Our framework utilizes CLIP with multi-knowledge enhanced learning, addressing long caption modeling limitations. It enables dynamic knowledge modeling between each image and its multiple corresponding captions, each capturing diverse aspects of crucial dermatological knowledge.
  • Figure 2: Overview of our MAKE framework. (a) Encoding multi-aspect clinical knowledge. (b) The process of aligning visual embeddings with multiple positive text embeddings. Knowledge 1: disease-aspect text; Knowledge 2: concept-aspect text. (c) A fine-grained alignment process, which matches each subtext embedding with knowledge-enhanced visual embeddings. (d) Diagnosis similarity-based weights that modulate alignment between subtexts and visual embeddings.