WARLearn: Weather-Adaptive Representation Learning

TL;DR

This paper introduces WARLearn, a novel framework designed for adaptive representation learning in challenging and adversarial weather conditions that surpasses the performance of state-of-the-art frameworks including FeatEnHancer, Image Adaptive YOLO, DENet, C2PNet, PairLIE and ZeroDCE by a substantial margin in adverse weather.

Abstract

This paper introduces WARLearn, a novel framework designed for adaptive representation learning in challenging and adversarial weather conditions. Leveraging the in-variance principal used in Barlow Twins, we demonstrate the capability to port the existing models initially trained on clear weather data to effectively handle adverse weather conditions. With minimal additional training, our method exhibits remarkable performance gains in scenarios characterized by fog and low-light conditions. This adaptive framework extends its applicability beyond adverse weather settings, offering a versatile solution for domains exhibiting variations in data distributions. Furthermore, WARLearn is invaluable in scenarios where data distributions undergo significant shifts over time, enabling models to remain updated and accurate. Our experimental findings reveal a remarkable performance, with a mean average precision (mAP) of 52.6% on unseen real-world foggy dataset (RTTS). Similarly, in low light conditions, our framework achieves a mAP of 55.7% on unseen real-world low light dataset (ExDark). Notably, WARLearn surpasses the performance of state-of-the-art frameworks including FeatEnHancer, Image Adaptive YOLO, DENet, C2PNet, PairLIE and ZeroDCE, by a substantial margin in adverse weather, improving the baseline performance in both foggy and low light conditions. The WARLearn code is available at https://github.com/ShubhamAgarwal12/WARLearn

Figures (5)

• Figure 1: Our proposed WARLearn framework initiates with the training of a model using clean weather data (a). Subsequently, leveraging the Barlow feature in-variance and redundancy reduction loss, we try to align the feature representations of adverse weather conditions with those obtained under clear weather conditions (b). This step enables the creation of a hybrid model, where the backbone is derived from the refined feature representations, while feature fusion and prediction components stem from the initial clean weather data (c). This hybrid model is then employed for robust predictions in adversarial weather conditions. In the figure, double-sided thick arrows indicate the flow of information between modules undergoing training in the respective stage. Dotted lines represent shared weights.
• Figure 2: In the Barlow Twins approach, we input two distorted versions of an image into the same feature extractor. This extractor is trained to align the features of the two distorted versions, fostering similarity in their representations. WARLearn employs a foundational feature extractor initially trained on clean data as a reference. It then trains a secondary feature extractor dedicated to distorted images, utilizing the foundational extractor as a reference. The objective is to match the features extracted from distorted images with those from corresponding clean images, thereby learning to mitigate the impact of adverse weather-induced distortions. The modules that undergo training using the Barlow redundancy reduction loss are highlighted in green colour. Broken boundaries indicate weight sharing between identical modules.
• Figure 3: Comparison of detection results between YOLOv3 baseline (left column) and WARLearn (right column) on the real-world foggy RTTS dataset. WARLearn recognizes more objects in foggy conditions with higher confidence scores.
• Figure 4: Comparison of detection results between YOLOv3 baseline (left column) and WARLearn (right column) on the real-world low-light ExDark dataset. WARLearn excels in recognizing more objects in low-light conditions with higher confidence scores.
• Figure 5: Detection results with WARLearn framework on an image with different levels of foggy degradation indicated by $\beta$ (\ref{['eq:beta']}) value below the image. These images are taken from simulated PascalVOC test foggy dataset.