WeCKD: Weakly-supervised Chained Distillation Network for Efficient Multimodal Medical Imaging

TL;DR

We address the challenge of limited labeled data in medical imaging by introducing WeCKD, a weakly-supervised chain-based knowledge distillation framework that transfers knowledge through a sequence of three interconnected models. Each stage trains on a small data subset while receiving softened predictions from its predecessor, guided by an attention-refinement module and temperature-annealed KD losses. Across six diverse datasets (otoscopic, MRI, microscopic), WeCKD achieves up to a +23.42% cumulative accuracy gain over a single backbone trained on the same limited data, approaching or matching fully supervised baselines in several cases without preprocessing. The approach demonstrates strong cross-modality generalization and efficiency, offering a practical path for deploying accurate medical image classifiers under labeling constraints, with future extensions to ViT backbones and unsupervised settings.

Abstract

Knowledge distillation (KD) has traditionally relied on a static teacher-student framework, where a large, well-trained teacher transfers knowledge to a single student model. However, these approaches often suffer from knowledge degradation, inefficient supervision, and reliance on either a very strong teacher model or large labeled datasets. To address these, we present the first-ever Weakly-supervised Chain-based KD network (WeCKD) that redefines knowledge transfer through a structured sequence of interconnected models. Unlike conventional KD, it forms a progressive distillation chain, where each model not only learns from its predecessor but also refines the knowledge before passing it forward. This structured knowledge transfer further enhances feature learning and addresses the limitations of one-step KD. Each model in the chain is trained on only a fraction of the dataset and shows that effective learning can be achieved with minimal supervision. Extensive evaluation on six imaging datasets across otoscopic, microscopic, and magnetic resonance imaging modalities shows that it generalizes and outperforms existing methods. Furthermore, the proposed distillation chain resulted in cumulative accuracy gains of up to +23% over a single backbone trained on the same limited data, which highlights its potential for real-world adoption.

Figures (5)

• Figure 1: Illustration of the proposed weakly-supervised knowledge distillation chain. The chain forms a weakly-supervised learning process by distilling knowledge progressively across three interconnected models ($M_1 \to M_2 \to M_3$), where each model learns from only 10% of the labeled data and refines the soft predictions received from its predecessor.
• Figure 2: Proposed extension to the model: the structural inclusion and exclusion of spatial-attention and feature-refinement modules within the backbone. The extension focuses on how these components enhance the model’s focus on discriminative regions while maintaining efficiency.
• Figure 3: Illustration of the multi-stage teacher–student relationships, where each subsequent model receives both hard labels and temperature-scaled soft logits from the previous stage.
• Figure 4: Visualization of the confusion matrices and AUC-ROC curves of the final distilled model $M_3$ for the six medical imaging datasets: (a) Brain Tumor; (b) Datos; (c) Eardrum; (d) EardrumDS; (e) Lung and Colon Cancer; and (f) Otomatch.
• Figure 5: Gradient-weighted class activation mapping visualization of an otoscopic image. The visualization highlights the spatial regions that most influence the model’s predictions, and it shows that the attention mechanism effectively aligns these activations with clinically relevant structures in the tympanic membrane.