SemiDDM-Weather: A Semi-supervised Learning Framework for All-in-one Adverse Weather Removal

TL;DR

This paper presents a pioneering semi-supervised all-in-one adverse weather removal framework built on the teacher-student network with a Denoising Diffusion Model (DDM) as the backbone, termed SemiDDM-Weather.

Abstract

Adverse weather removal aims to restore clear vision under adverse weather conditions. Existing methods are mostly tailored for specific weather types and rely heavily on extensive labeled data. In dealing with these two limitations, this paper presents a pioneering semi-supervised all-in-one adverse weather removal framework built on the teacher-student network with a Denoising Diffusion Model (DDM) as the backbone, termed SemiDDM-Weather. As for the design of DDM backbone in our SemiDDM-Weather, we adopt the SOTA Wavelet Diffusion Model-Wavediff with customized inputs and loss functions, devoted to facilitating the learning of many-to-one mapping distributions for efficient all-in-one adverse weather removal with limited label data. To mitigate the risk of misleading model training due to potentially inaccurate pseudo-labels generated by the teacher network in semi-supervised learning, we introduce quality assessment and content consistency constraints to screen the "optimal" outputs from the teacher network as the pseudo-labels, thus more effectively guiding the student network training with unlabeled data. Experimental results show that on both synthetic and real-world datasets, our SemiDDM-Weather consistently delivers high visual quality and superior adverse weather removal, even when compared to fully supervised competitors. Our code and pre-trained model are available at this repository.

Figures (10)

• Figure 1: The denoising process of WaveDiff. The operations within the dashed box will be iteratively repeated until obtaining the final denoised output $q(y_0')$
• Figure 2: Illustration of our SemiDDM-Weather framework, which innovatively integrates a DDM into the teacher-student network
• Figure 3: The schematic diagram of labeled data training in the student network
• Figure 4: The schematic diagram of the dense regular grid cropping strategy. In this diagram, the grid size represents the spacing used for cropping, while the box size represents the size of the cropped images. In our method, we set the spacing to 4 pixels, and each crop is sized at $64\times64$ pixels. The cropping sequence systematically proceeds along the grid from left to right and from top to bottom, and the cropped patches of $x$ are collected in a dictionary $\mathcal{P} = \{{x}_{p_1}, {x}_{p_2}, ..., {x}_{p_K}\}, {x}_{p_k}={\text{Crop}}_{64}(\mathbf{M}_{k} \circ x)$, where $\circ$ indicates element-wise multiple and $\mathbf{M}_{k}$ is the $k^{th}$ patch mask of $x$ (1 for corresponding patch locations while 0 for others)
• Figure 5: The schematic diagram of the model inference process. For the denoising process, please refer to Fig. \ref{['fig:sample']} for a similar process. The cropping process can similarly reference Fig. \ref{['fig:crop']}