Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IKD+: Reliable Low Complexity Deep Models For Retinopathy Classification

Shreyas Bhat Brahmavar, Rohit Rajesh, Tirtharaj Dash, Lovekesh Vig, Tanmay Tulsidas Verlekar, Md Mahmudul Hasan, Tariq Khan, Erik Meijering, Ashwin Srinivasan

TL;DR

Diabetic retinopathy classifiers often achieve high accuracy but rely on large, resource-intensive networks with limited calibration reliability in clinical use. The authors propose IKD+, a calibration-sensitive iterative knowledge distillation framework that progressively compresses a heavy teacher into smaller students while enforcing reliability through calibration losses. They evaluate two calibration methods, Temperature Scaling and Platt-scaling, on an EfficientNet ensemble, achieving up to 500-fold parameter reduction with only about 2% accuracy loss and calibration equal to or better than the original. Experiments on the Retinal Fundus Multi-Disease Image Dataset show robust performance across base models (EfficientNet-B6 and -B5) with substantial size savings and improved reliability. The work offers a practical route to deploy reliable, low-complexity retinopathy classifiers in real-world clinical settings.

Abstract

Deep neural network (DNN) models for retinopathy have estimated predictive accuracies in the mid-to-high 90%. However, the following aspects remain unaddressed: State-of-the-art models are complex and require substantial computational infrastructure to train and deploy; The reliability of predictions can vary widely. In this paper, we focus on these aspects and propose a form of iterative knowledge distillation(IKD), called IKD+ that incorporates a tradeoff between size, accuracy and reliability. We investigate the functioning of IKD+ using two widely used techniques for estimating model calibration (Platt-scaling and temperature-scaling), using the best-performing model available, which is an ensemble of EfficientNets with approximately 100M parameters. We demonstrate that IKD+ equipped with temperature-scaling results in models that show up to approximately 500-fold decreases in the number of parameters than the original ensemble without a significant loss in accuracy. In addition, calibration scores (reliability) for the IKD+ models are as good as or better than the base mode

IKD+: Reliable Low Complexity Deep Models For Retinopathy Classification

TL;DR

Diabetic retinopathy classifiers often achieve high accuracy but rely on large, resource-intensive networks with limited calibration reliability in clinical use. The authors propose IKD+, a calibration-sensitive iterative knowledge distillation framework that progressively compresses a heavy teacher into smaller students while enforcing reliability through calibration losses. They evaluate two calibration methods, Temperature Scaling and Platt-scaling, on an EfficientNet ensemble, achieving up to 500-fold parameter reduction with only about 2% accuracy loss and calibration equal to or better than the original. Experiments on the Retinal Fundus Multi-Disease Image Dataset show robust performance across base models (EfficientNet-B6 and -B5) with substantial size savings and improved reliability. The work offers a practical route to deploy reliable, low-complexity retinopathy classifiers in real-world clinical settings.

Abstract

Deep neural network (DNN) models for retinopathy have estimated predictive accuracies in the mid-to-high 90%. However, the following aspects remain unaddressed: State-of-the-art models are complex and require substantial computational infrastructure to train and deploy; The reliability of predictions can vary widely. In this paper, we focus on these aspects and propose a form of iterative knowledge distillation(IKD), called IKD+ that incorporates a tradeoff between size, accuracy and reliability. We investigate the functioning of IKD+ using two widely used techniques for estimating model calibration (Platt-scaling and temperature-scaling), using the best-performing model available, which is an ensemble of EfficientNets with approximately 100M parameters. We demonstrate that IKD+ equipped with temperature-scaling results in models that show up to approximately 500-fold decreases in the number of parameters than the original ensemble without a significant loss in accuracy. In addition, calibration scores (reliability) for the IKD+ models are as good as or better than the base mode
Paper Structure (9 sections, 4 equations, 2 figures, 4 tables, 1 algorithm)

This paper contains 9 sections, 4 equations, 2 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Major contributions
  4. Reliable IKD (IKD+)
  5. Empirical Evaluation
  6. Dataset and Machines specifications
  7. Method
  8. Results
  9. CONCLUSION

Figures (2)

  • Figure 1: Distribution of the Dataset.
  • Figure 2: Bar plots indicating the probability of classifying an image as diabetic retinopathy (DR), laser scar (LS) and tortuous (TV) using baseline and IKD+ model.