Pseudo-Label Guided Real-World Image De-weathering: A Learning Framework with Imperfect Supervision

TL;DR

This work tackles real-world image de-weathering under imperfect supervision caused by misaligned ground-truth pairs. It introduces a pseudo-label guided learning framework comprising a Consistent Label Constructor (CLC), a Cross-frame Similarity Aggregation (CSA) module, and an Information Allocation Strategy (IAS) to jointly leverage pseudo-labels and original labels. The CSA-CLC extension integrates cross-frame self-similarity using Graph Attention Networks to produce high-quality pseudo-labels, which are then fused with original GT signals during training. Extensive experiments on GT-RAIN-SNOW and WeatherStream demonstrate superior de-weathering performance and ablation studies validate the effectiveness of CSA, frame utilization, and supervision strategies. The approach reduces the need for perfectly aligned real-world data and enables robust restoration across multiple weather conditions with practical impact for vision systems exposed to adverse weather.

Abstract

Real-world image de-weathering aims at removingvarious undesirable weather-related artifacts, e.g., rain, snow,and fog. To this end, acquiring ideal training pairs is crucial.Existing real-world datasets are typically constructed paired databy extracting clean and degraded images from live streamsof landscape scene on the Internet. Despite the use of strictfiltering mechanisms during collection, training pairs inevitablyencounter inconsistency in terms of lighting, object position, scenedetails, etc, making de-weathering models possibly suffer fromdeformation artifacts under non-ideal supervision. In this work,we propose a unified solution for real-world image de-weatheringwith non-ideal supervision, i.e., a pseudo-label guided learningframework, to address various inconsistencies within the realworld paired dataset. Generally, it consists of a de-weatheringmodel (De-W) and a Consistent Label Constructor (CLC), bywhich restoration result can be adaptively supervised by originalground-truth image to recover sharp textures while maintainingconsistency with the degraded inputs in non-weather contentthrough the supervision of pseudo-labels. Particularly, a Crossframe Similarity Aggregation (CSA) module is deployed withinCLC to enhance the quality of pseudo-labels by exploring thepotential complementary information of multi-frames throughgraph model. Moreover, we introduce an Information AllocationStrategy (IAS) to integrate the original ground-truth imagesand pseudo-labels, thereby facilitating the joint supervision forthe training of de-weathering model. Extensive experimentsdemonstrate that our method exhibits significant advantageswhen trained on imperfectly aligned de-weathering datasets incomparison with other approaches.

Figures (10)

• Figure 1: Examples of misalignment categories between the degraded images (top) and the original GT (bottom). These instances are derived from two real-world de-weathering datasets. (i.e., GT-Rain-Snow ba2022not and WeatherStream zhang2023weatherstream). The pairs in (b) and (c) show inconsistencies in color and lighting. (a) and (d) indicate misalignment caused by object motion, eg., moving vehicle, swaying leaves. The inconsistent contents are highlighted in red boxes.
• Figure 2: For the query patch $\mathbf{p}_{t}$ in the current frame $\mathbf{I}_{t}$ being processed, its matched patches $\mathbf{p}_{t-1}$ in the neighboring frame $\mathbf{I}_{t-1}$ that have similar textures to the query patch may not be obscured by snowflakes. Each patch's left image represents the feature map obtained from feature extractor VGG19. $\mathbf{p}_{t-1}$ exhibits smaller feature-space distances to the ground truth and can serve as supplementary information.
• Figure 3: The variation in mutual information during training phase. We compute the mutual information between the output and both the target and the input. The model is trained on GT-Rain-Snow ba2022not dataset for a total of $20$ epochs, with mutual information being calculated every $2$ epochs.
• Figure 4: The pipeline of proposed method. A Consistent Label Constructor (CLC) is pre-trained to generate a pseudo-label $\mathbf{\tilde{I}}$ from multiple degraded frames. Then both the original label $\mathbf{I}_{gt}$ and the pseudo-label $\mathbf{\tilde{I}}$ are employed to regulate the output $\mathbf{\hat{I}}$ of the De-weathering Model (De-W). For testing, solely the De-W is utilized to restore the weather degraded input.
• Figure 5: Details of the Cross-frame Similarity Aggregation (CSA) module, including Flexible Cross-frame Matching (FCM) and Non-local Feature Aggregation (NFA).