MWFormer: Multi-Weather Image Restoration Using Degradation-Aware Transformers

TL;DR

A multi-weather Transformer that aims to solve multiple weather-induced degradations using a single, unified architecture, and allows for a novel way of tuning, during application, to either a single type of weather restoration or to hybrid weather restoration without any retraining, offering greater controllability than existing methods.

Abstract

Restoring images captured under adverse weather conditions is a fundamental task for many computer vision applications. However, most existing weather restoration approaches are only capable of handling a specific type of degradation, which is often insufficient in real-world scenarios, such as rainy-snowy or rainy-hazy weather. Towards being able to address these situations, we propose a multi-weather Transformer, or MWFormer for short, which is a holistic vision Transformer that aims to solve multiple weather-induced degradations using a single, unified architecture. MWFormer uses hyper-networks and feature-wise linear modulation blocks to restore images degraded by various weather types using the same set of learned parameters. We first employ contrastive learning to train an auxiliary network that extracts content-independent, distortion-aware feature embeddings that efficiently represent predicted weather types, of which more than one may occur. Guided by these weather-informed predictions, the image restoration Transformer adaptively modulates its parameters to conduct both local and global feature processing, in response to multiple possible weather. Moreover, MWFormer allows for a novel way of tuning, during application, to either a single type of weather restoration or to hybrid weather restoration without any retraining, offering greater controllability than existing methods. Our experimental results on multi-weather restoration benchmarks show that MWFormer achieves significant performance improvements compared to existing state-of-the-art methods, without requiring much computational cost. Moreover, we demonstrate that our methodology of using hyper-networks can be integrated into various network architectures to further boost their performance. The code is available at: https://github.com/taco-group/MWFormer

Figures (16)

• Figure 1: Top row: Comparison of the MWFormer architecture with those of existing task-specific networks. Bottom: Restoration performance and computational cost of three versions of MWFormer having different numbers of channels against three competitive multi-weather restoration models. MWFormer achieves generally better performance with 100$\times$ less computation than the SOTA model WeatherDiffusion Diffusion.
• Figure 2: The architecture of MWFormer. The main image processing network consists of a Transformer encoder, a Transformer decoder, and convolution tails. (a) A feature extraction network learns to generate some of the parameters of the Transformer blocks and intra-patch Transformer blocks in the main network, thereby partially controlling the production of intermediate feature maps. (b) The Transformer block in the encoder of the main network, which is guided by the feature vector. (c) Transformer decoder of the main network, whose queries are learnable parameters.
• Figure 3: Comparison of the default architecture with two test-time variants applied in special cases. To conduct a single weather-type restoration, the feature extraction network is replaced by a fixed feature vector. To conduct hybrid weather restoration, the image processing network is cascaded to remove degradations sequentially, stage by stage.
• Figure 4: Qualitative comparisons on the RainDrop Raindrop test set. MWFormer effectively removed raindrop artifacts under various scenarios, yielding output images with either fewer shadows or less blur than the other compared models.
• Figure 5: Visual comparisons on the Test1 OutdoorRain (rain+fog) set. MWFormer performed the best on both detail restoration and luminance retention. AirNet failed to remove most of the degradations, TransWeather recovered fewer details, and the WeatherDiffusion introduced color distortions.