Continual All-in-One Adverse Weather Removal with Knowledge Replay on a Unified Network Structure

TL;DR

A novel continual learning framework with effective knowledge replay (KR) on a unified network structure is developed, which performs competitively to existing dedicated or joint training image restoration methods.

Abstract

In real-world applications, image degeneration caused by adverse weather is always complex and changes with different weather conditions from days and seasons. Systems in real-world environments constantly encounter adverse weather conditions that are not previously observed. Therefore, it practically requires adverse weather removal models to continually learn from incrementally collected data reflecting various degeneration types. Existing adverse weather removal approaches, for either single or multiple adverse weathers, are mainly designed for a static learning paradigm, which assumes that the data of all types of degenerations to handle can be finely collected at one time before a single-phase learning process. They thus cannot directly handle the incremental learning requirements. To address this issue, we made the earliest effort to investigate the continual all-in-one adverse weather removal task, in a setting closer to real-world applications. Specifically, we develop a novel continual learning framework with effective knowledge replay (KR) on a unified network structure. Equipped with a principal component projection and an effective knowledge distillation mechanism, the proposed KR techniques are tailored for the all-in-one weather removal task. It considers the characteristics of the image restoration task with multiple degenerations in continual learning, and the knowledge for different degenerations can be shared and accumulated in the unified network structure. Extensive experimental results demonstrate the effectiveness of the proposed method to deal with this challenging task, which performs competitively to existing dedicated or joint training image restoration methods. Our code is available at https://github.com/xiaojihh/CL_all-in-one.

Figures (7)

• Figure 1: Illustration of the proposed continual learning for all-in-one adverse weather removal task. Left: Model trained on hazy images can only conduct dehazing; Middle: When the model is trained continually on rainy images without accessing previous data, it can well conduct dehazing and deraining simultaneously; Right: As continual learning goes on, the model can accumulate knowledge towards an all-in-one model, to deal with all types of adverse weather.
• Figure 2: The proposed continual learning framework with knowledge replay on a unified network structure for all-in-one adverse weather removal. The framework mainly includes three components: the backbone network and training scheme applied for single adverse weather removal; effective knowledge replay based distillation on the network prediction; and the principal component projection in knowledge replay.
• Figure 3: Detail architecture of the principal component projection.
• Figure 4: Feature map visualization of intermediate features extracted by $\mathcal{F}$ in (b), and the principal component encoder $\psi(\mathcal{F})$ in (c).
• Figure 5: Visualization comparison of adverse weather removal using different continual learning algorithms.