WM-MoE: Weather-aware Multi-scale Mixture-of-Experts for Blind Adverse Weather Removal

TL;DR

The paper tackles blind adverse weather removal, where weather type and intensity are unknown, by introducing WM-MoE, a Transformer-based framework that combines a Weather-aware Router (WEAR), Multi-Scale Experts (MSE), and Weather Guidance Fine-grained Contrastive Learning (WGF-CL). WEAR decouples content and weather cues to route image tokens to weather-specific experts, while MSE enables multi-scale feature fusion to handle varying weather conditions. WGF-CL learns discriminative weather representations using patch-level positives from the same weather, guiding robust weather-aware routing. The approach achieves state-of-the-art results on MAW-Sim and All-Weather datasets and demonstrates benefits for downstream segmentation, highlighting its practical impact for autonomous driving under diverse weather.

Abstract

Adverse weather removal tasks like deraining, desnowing, and dehazing are usually treated as separate tasks. However, in practical autonomous driving scenarios, the type, intensity,and mixing degree of weather are unknown, so handling each task separately cannot deal with the complex practical scenarios. In this paper, we study the blind adverse weather removal problem. Mixture-of-Experts (MoE) is a popular model that adopts a learnable gate to route the input to different expert networks. The principle of MoE involves using adaptive networks to process different types of unknown inputs. Therefore, MoE has great potential for blind adverse weather removal. However, the original MoE module is inadequate for coupled multiple weather types and fails to utilize multi-scale features for better performance. To this end, we propose a method called Weather-aware Multi-scale MoE (WM-MoE) based on Transformer for blind weather removal. WM-MoE includes two key designs: WEather-Aware Router (WEAR) and Multi-Scale Experts (MSE). WEAR assigns experts for each image token based on decoupled content and weather features, which enhances the model's capability to process multiple adverse weathers. To obtain discriminative weather features from images, we propose Weather Guidance Fine-grained Contrastive Learning (WGF-CL), which utilizes weather cluster information to guide the assignment of positive and negative samples for each image token. Since processing different weather types requires different receptive fields, MSE leverages multi-scale features to enhance the spatial relationship modeling capability, facilitating the high-quality restoration of diverse weather types and intensities. Our method achieves state-of-the-art performance in blind adverse weather removal on two public datasets and our dataset. We also demonstrate the advantage of our method on downstream segmentation tasks.

Figures (11)

• Figure 1: Comparison between (a) naive Mixture-of-Experts (MoE) module lepikhin2020moetransformer and (b) our proposed Weather-aware Multi-scale Mixture-of-Experts (WM-MoE) module. The linear layer router and point-wise FFN experts in the original MoE fail to deal with coupled content and weather features, and local information. Respectfully, we propose a Weather-aware Router to select experts dynamically based on decoupled content and weather embedding and Multi-scale Experts to make full use of local and multi-scale features.
• Figure 2: Comparison of normalized routing scores histogram between MoE and Weather-aware Router (ours).
• Figure 3: Comparison of t-SNE of routing weights between MoE and Weather-aware Router (ours).
• Figure 4: Comparison of ViT dosovitskiy2020vit, PVT wang2021pvt, Shunted Transformer ren2022shunted, MoE lepikhin2020gshard and proposed Weather-aware Multi-scale MoE in terms of FFN module.
• Figure 5: The illustration of proposed Weather Guidance Fine-grained Contrastive Learning. Patch embeddings from images with the same weather are regarded as positive samples and from different weather are negative samples.