Multi-Weather Image Restoration via Histogram-Based Transformer Feature Enhancement

TL;DR

The paper tackles the problem of restoring images degraded by mixed weather conditions, where single-weather models struggle due to unknown degradation mixtures modeled by expressions such as $I(x)=J(x)t(x)+A(1-t(x))$. It introduces a Histogram Transformer backbone combined with a Task Intra-Patch Block and a Task Sequence Generator to extract and leverage multi-scale, task-specific degradation features, and employs adaptive upsampling to fuse task cues with image features. Key contributions include the Task Sequence Generator for stage-aware degradation features, the Histogram Transformer Block with Dynamic-range Histogram Self-Attention, and an Adaptive Mixup fusion strategy, all validated by state-of-the-art results on multiple public datasets and improved object-detection performance on restored imagery. The approach provides a practical, deployable solution for robust multi-weather restoration with potential impact on autonomous driving and other vision-heavy applications where weather-induced degradation is diverse and dynamic.

Abstract

Currently, the mainstream restoration tasks under adverse weather conditions have predominantly focused on single-weather scenarios. However, in reality, multiple weather conditions always coexist and their degree of mixing is usually unknown. Under such complex and diverse weather conditions, single-weather restoration models struggle to meet practical demands. This is particularly critical in fields such as autonomous driving, where there is an urgent need for a model capable of effectively handling mixed weather conditions and enhancing image quality in an automated manner. In this paper, we propose a Task Sequence Generator module that, in conjunction with the Task Intra-patch Block, effectively extracts task-specific features embedded in degraded images. The Task Intra-patch Block introduces an external learnable sequence that aids the network in capturing task-specific information. Additionally, we employ a histogram-based transformer module as the backbone of our network, enabling the capture of both global and local dynamic range features. Our proposed model achieves state-of-the-art performance on public datasets.

Figures (8)

• Figure 1: The proposed method and example results of TransWeatherJeyaMariaJoseValanarasu2022TransWeatherTR and TTAMSwen2024multipleweatherimagesrestoration for multi-weather image restoration.
• Figure 2: Overview of the proposed network. The degraded image is progressively processed through Histogram Transformer Blocks and Task Intra-PT Blocks. The degradation features and task features obtained from these blocks are fused using the SPFI block. The multi-level task features are then passed through the Task Sequence Generator to produce a task sequence. Finally, the clean image is restored through adaptive upsampling.
• Figure 3: Detail of Task Transformer Block. By calculating the q introduced from the outside and the kv generated by the image, the attention map is input to the multi-layer perceptron to obtain the feature map with task information.
• Figure 4: Qualitative results comparison of the proposed method with existing state-of-the-art methods All-in-OneRuiQian2017AttentiveGA , TransWeatherJeyaMariaJoseValanarasu2022TransWeatherTR, and TTAMSwen2024multipleweatherimagesrestoration on Raindrop datasetRuiQian2017AttentiveGA for raindrop removal.
• Figure 5: Qualitative results comparison of the proposed method with existing state-of-the-art methods All-in-OneRuiQian2017AttentiveGA , TransWeatherJeyaMariaJoseValanarasu2022TransWeatherTR and TTAMSwen2024multipleweatherimagesrestoration, on Test1 datasetLiRuoteng2019HeavyRI for rain removal and haze removal.