Leveraging Knowledge Distillation for Lightweight Skin Cancer Classification: Balancing Accuracy and Computational Efficiency

TL;DR

This work tackles the need for accurate skin-cancer classification on resource-limited devices by combining a high-capacity, fusion-based teacher with a compact, distillation-guided student. A six-stage image preprocessing pipeline, data augmentation, and a 16-bit quantization step enable edge deployment of a 469.77 KB classifier that maintains near-perfect accuracy on HAM10000 and Kaggle benchmarks. The approach demonstrates that knowledge distillation, when coupled with careful preprocessing and model compression, can substantially improve both performance and efficiency over existing lightweight methods. The findings have practical implications for accessible, on-device skin cancer screening in low-resource settings, with potential for multiclass expansion and explainable AI integration in future work.

Abstract

Skin cancer is a major concern to public health, accounting for one-third of the reported cancers. If not detected early, the cancer has the potential for severe consequences. Recognizing the critical need for effective skin cancer classification, we address the limitations of existing models, which are often too large to deploy in areas with limited computational resources. In response, we present a knowledge distillation based approach for creating a lightweight yet high-performing classifier. The proposed solution involves fusing three models, namely ResNet152V2, ConvNeXtBase, and ViT Base, to create an effective teacher model. The teacher model is then employed to guide a lightweight student model of size 2.03 MB. This student model is further compressed to 469.77 KB using 16-bit quantization, enabling smooth incorporation into edge devices. With six-stage image preprocessing, data augmentation, and a rigorous ablation study, the model achieves an impressive accuracy of 98.75% on the HAM10000 dataset and 98.94% on the Kaggle dataset in classifying benign and malignant skin cancers. With its high accuracy and compact size, our model appears to be a potential choice for accurate skin cancer classification, particularly in resource-constrained settings.

Figures (8)

• Figure 1: Overview of the proposed method.
• Figure 2: Image preprocessing steps.
• Figure 3: Changes in the input image after each preprocessing step.
• Figure 4: Teacher model used for knowledge distillation.
• Figure 5: Student model used for classification.