Multiple weather images restoration using the task transformer and adaptive mixup strategy

TL;DR

The paper tackles restoration of images degraded by mixed weather conditions (rain, haze, snow) in autonomous-driving contexts. It introduces a multi-task restoration network that leverages a Task Intra-patch Block, a Task Sequence Generator, Fast Fourier Convolution for global context, and Adaptive Mixup for texture-preserving feature fusion. The approach demonstrates state-of-the-art performance on public datasets and improves downstream object detection, particularly in large degraded regions. By learning task-specific degradation sequences and incorporating global information, the method offers a practical path toward robust perception under adverse weather.

Abstract

The current state-of-the-art in severe weather removal predominantly focuses on single-task applications, such as rain removal, haze removal, and snow removal. However, real-world weather conditions often consist of a mixture of several weather types, and the degree of weather mixing in autonomous driving scenarios remains unknown. In the presence of complex and diverse weather conditions, a single weather removal model often encounters challenges in producing clear images from severe weather images. Therefore, there is a need for the development of multi-task severe weather removal models that can effectively handle mixed weather conditions and improve image quality in autonomous driving scenarios. In this paper, we introduce a novel multi-task severe weather removal model that can effectively handle complex weather conditions in an adaptive manner. Our model incorporates a weather task sequence generator, enabling the self-attention mechanism to selectively focus on features specific to different weather types. To tackle the challenge of repairing large areas of weather degradation, we introduce Fast Fourier Convolution (FFC) to increase the receptive field. Additionally, we propose an adaptive upsampling technique that effectively processes both the weather task information and underlying image features by selectively retaining relevant information. Our proposed model has achieved state-of-the-art performance on the publicly available dataset.

Figures (5)

• Figure 1: Overview of the proposed network. The degraded image will first be input to the deconvolution module for preliminary processing, and then the task features will be extracted layer by layer through the TIPB module and the transformer module, and the task features will be input to the Task Sequence Generator to generate a task sequence. Finally, the background features from the bottom layer will be combined with the task features through the FFC module to restore a clear image.
• Figure 2: Detail of Task Transformer Block. By calculating the q introduced from the outside and the kv generated by the image, the attention map is input to the multi-layer perceptron to obtain the feature map with task information.
• Figure 3: A comparison of the intermediate results of three tasks using the TransWeatherJeyaMariaJoseValanarasu2022TransWeatherTR Model and the Task-sequence Generator module. (a) and (b) is the output result of removing rain, (c) and (d) is the output result of removing rain and haze, and (e) is the output result of removing snow.
• Figure 4: Detail of Task FFC Block. The local branch utilizes conventional convolution operations, while the global branch employs channel-wise fast Fourier transform to capture the broader context of the image.
• Figure 5: Qualitative results comparison of the proposed method with existing state-of-the-art methods All-in-OneRuiQian2017AttentiveGA and TransWeatherJeyaMariaJoseValanarasu2022TransWeatherTR, on Raindrop datasetRuiQian2017AttentiveGA for raindrop removal.