Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowledge Distillation for Image Restoration : Simultaneous Learning from Degraded and Clean Images

Yongheng Zhang, Danfeng Yan

TL;DR

The paper tackles the challenge of compressing image restoration models without sacrificing performance. It introduces Simultaneous Learning Knowledge Distillation (SLKD), a dual-teacher, single-student framework with Degradation Removal Learning (DRL) and Image Reconstruction Learning (IRL), guided by BRISQUE- and PIQE-based feature extractors and a Scale Conversion Module to learn from both degraded and clean inputs. Key contributions include a novel dual-teacher distillation scheme, feature-level losses L_NC and L_ET, and a joint image/pixel/ KL objective that yields substantial reductions in FLOPs and parameters (over 80%) while maintaining strong restoration performance across five datasets and three tasks (deraining, deblurring, denoising). Ablation studies validate the necessity of DRL, IRL, and the two feature-level losses. The approach enables efficient deployment of high-quality image restoration on resource-constrained platforms, offering a practical path for compact yet capable restoration models.

Abstract

Model compression through knowledge distillation has seen extensive application in classification and segmentation tasks. However, its potential in image-to-image translation, particularly in image restoration, remains underexplored. To address this gap, we propose a Simultaneous Learning Knowledge Distillation (SLKD) framework tailored for model compression in image restoration tasks. SLKD employs a dual-teacher, single-student architecture with two distinct learning strategies: Degradation Removal Learning (DRL) and Image Reconstruction Learning (IRL), simultaneously. In DRL, the student encoder learns from Teacher A to focus on removing degradation factors, guided by a novel BRISQUE extractor. In IRL, the student decoder learns from Teacher B to reconstruct clean images, with the assistance of a proposed PIQE extractor. These strategies enable the student to learn from degraded and clean images simultaneously, ensuring high-quality compression of image restoration models. Experimental results across five datasets and three tasks demonstrate that SLKD achieves substantial reductions in FLOPs and parameters, exceeding 80\%, while maintaining strong image restoration performance.

Knowledge Distillation for Image Restoration : Simultaneous Learning from Degraded and Clean Images

TL;DR

The paper tackles the challenge of compressing image restoration models without sacrificing performance. It introduces Simultaneous Learning Knowledge Distillation (SLKD), a dual-teacher, single-student framework with Degradation Removal Learning (DRL) and Image Reconstruction Learning (IRL), guided by BRISQUE- and PIQE-based feature extractors and a Scale Conversion Module to learn from both degraded and clean inputs. Key contributions include a novel dual-teacher distillation scheme, feature-level losses L_NC and L_ET, and a joint image/pixel/ KL objective that yields substantial reductions in FLOPs and parameters (over 80%) while maintaining strong restoration performance across five datasets and three tasks (deraining, deblurring, denoising). Ablation studies validate the necessity of DRL, IRL, and the two feature-level losses. The approach enables efficient deployment of high-quality image restoration on resource-constrained platforms, offering a practical path for compact yet capable restoration models.

Abstract

Model compression through knowledge distillation has seen extensive application in classification and segmentation tasks. However, its potential in image-to-image translation, particularly in image restoration, remains underexplored. To address this gap, we propose a Simultaneous Learning Knowledge Distillation (SLKD) framework tailored for model compression in image restoration tasks. SLKD employs a dual-teacher, single-student architecture with two distinct learning strategies: Degradation Removal Learning (DRL) and Image Reconstruction Learning (IRL), simultaneously. In DRL, the student encoder learns from Teacher A to focus on removing degradation factors, guided by a novel BRISQUE extractor. In IRL, the student decoder learns from Teacher B to reconstruct clean images, with the assistance of a proposed PIQE extractor. These strategies enable the student to learn from degraded and clean images simultaneously, ensuring high-quality compression of image restoration models. Experimental results across five datasets and three tasks demonstrate that SLKD achieves substantial reductions in FLOPs and parameters, exceeding 80\%, while maintaining strong image restoration performance.
Paper Structure (10 sections, 6 equations, 4 figures, 4 tables)

This paper contains 10 sections, 6 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Overall Architecture
  4. Degradation Removal Learning
  5. Image Reconstruction Learning
  6. Overall Loss
  7. Experiments and Analysis
  8. Comparisons with State-of-the-arts
  9. Ablation Studies
  10. Conclusion

Figures (4)

  • Figure 1: Qualitative comparison with knowledge distillation methods across image deraining, deblurring and denoising.
  • Figure 2: The overall architecture of our proposed Simultaneous Learning Knowledge Distillation (SLKD) strategy for image restoration. SLKD includes a Degradation Removal Learning (DRL) and an Image Reconstruction Learning (IRL).
  • Figure 3: Qualitative comparison with light-weight methods.
  • Figure 4: Qualitative ablation study results on Rain1400fu2017removing.