Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data-free Distillation with Degradation-prompt Diffusion for Multi-weather Image Restoration

Pei Wang, Xiaotong Luo, Yuan Xie, Yanyun Qu

TL;DR

MWIR restoration under adverse weather suffers from data scarcity and heavy models. The paper presents D4IR, a data-free distillation framework that uses a degradation-aware prompt adapter (DPA) and content-driven diffusion (CCD) to synthesize domain-relevant degraded images from unpaired web data, enabling pixel-wise knowledge distillation (PKD) from a pre-trained teacher without original training data. D4IR replaces GAN-based pseudo-data generation with diffusion-based synthesis, mitigating mode collapse and domain shift, and demonstrates competitive performance against data-based distillation and strong unsupervised methods across deraining, dehazing, and desnowing on Rain100L, SOTS, and Snow100K. The approach reduces data requirements and is suitable for memory-limited devices, potentially broadening practical deployment of MWIR systems. Future work may further improve prompt generation and diffusion conditioning.

Abstract

Multi-weather image restoration has witnessed incredible progress, while the increasing model capacity and expensive data acquisition impair its applications in memory-limited devices. Data-free distillation provides an alternative for allowing to learn a lightweight student model from a pre-trained teacher model without relying on the original training data. The existing data-free learning methods mainly optimize the models with the pseudo data generated by GANs or the real data collected from the Internet. However, they inevitably suffer from the problems of unstable training or domain shifts with the original data. In this paper, we propose a novel Data-free Distillation with Degradation-prompt Diffusion framework for multi-weather Image Restoration (D4IR). It replaces GANs with pre-trained diffusion models to avoid model collapse and incorporates a degradation-aware prompt adapter to facilitate content-driven conditional diffusion for generating domain-related images. Specifically, a contrast-based degradation prompt adapter is firstly designed to capture degradation-aware prompts from web-collected degraded images. Then, the collected unpaired clean images are perturbed to latent features of stable diffusion, and conditioned with the degradation-aware prompts to synthesize new domain-related degraded images for knowledge distillation. Experiments illustrate that our proposal achieves comparable performance to the model distilled with original training data, and is even superior to other mainstream unsupervised methods.

Data-free Distillation with Degradation-prompt Diffusion for Multi-weather Image Restoration

TL;DR

MWIR restoration under adverse weather suffers from data scarcity and heavy models. The paper presents D4IR, a data-free distillation framework that uses a degradation-aware prompt adapter (DPA) and content-driven diffusion (CCD) to synthesize domain-relevant degraded images from unpaired web data, enabling pixel-wise knowledge distillation (PKD) from a pre-trained teacher without original training data. D4IR replaces GAN-based pseudo-data generation with diffusion-based synthesis, mitigating mode collapse and domain shift, and demonstrates competitive performance against data-based distillation and strong unsupervised methods across deraining, dehazing, and desnowing on Rain100L, SOTS, and Snow100K. The approach reduces data requirements and is suitable for memory-limited devices, potentially broadening practical deployment of MWIR systems. Future work may further improve prompt generation and diffusion conditioning.

Abstract

Multi-weather image restoration has witnessed incredible progress, while the increasing model capacity and expensive data acquisition impair its applications in memory-limited devices. Data-free distillation provides an alternative for allowing to learn a lightweight student model from a pre-trained teacher model without relying on the original training data. The existing data-free learning methods mainly optimize the models with the pseudo data generated by GANs or the real data collected from the Internet. However, they inevitably suffer from the problems of unstable training or domain shifts with the original data. In this paper, we propose a novel Data-free Distillation with Degradation-prompt Diffusion framework for multi-weather Image Restoration (D4IR). It replaces GANs with pre-trained diffusion models to avoid model collapse and incorporates a degradation-aware prompt adapter to facilitate content-driven conditional diffusion for generating domain-related images. Specifically, a contrast-based degradation prompt adapter is firstly designed to capture degradation-aware prompts from web-collected degraded images. Then, the collected unpaired clean images are perturbed to latent features of stable diffusion, and conditioned with the degradation-aware prompts to synthesize new domain-related degraded images for knowledge distillation. Experiments illustrate that our proposal achieves comparable performance to the model distilled with original training data, and is even superior to other mainstream unsupervised methods.
Paper Structure (16 sections, 9 equations, 5 figures, 5 tables)

This paper contains 16 sections, 9 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multi-weather Image Restoration
  4. Data-free Knowledge Distillation
  5. Conditional Diffusion Models
  6. Proposed Method
  7. Preliminary
  8. Method Overview
  9. Degradation-aware Prompt Adapter
  10. Content-driven Conditional Diffusion
  11. Pixel-wise Knowledge Distillation
  12. Experiments
  13. Experimental Settings
  14. Comparisons with the State-of-the-art
  15. Ablation Studies
  16. ...and 1 more sections

Figures (5)

  • Figure 1: The schematic diagram comparison of the data-free distillation methods for MWIR. (a) GAN-based methods : directly map the pure noise to the original data domain, while (b) our diffusion-based method synthesizes images with separate content and degradation information.
  • Figure 2: The overall framework of our proposed D4IR. It separately extracts degradation-aware and content-related features from unpaired web-collected images to guide SD in synthesizing the source domain related images for knowledge distillation.
  • Figure 3: Visual comparisons of D4IR and other methods for image deraining on Rain100L. Zoom in for a better view.
  • Figure 4: Visual comparisons of D4IR and other methods for image dehazing on SOTS. Zoom in for a better view.
  • Figure 5: Visualized samples synthesized by DFMC (Top), SD (Middle), and our D4IR (Bottom) for image dehazing.