Sharing the Learned Knowledge-base to Estimate Convolutional Filter Parameters for Continual Image Restoration

TL;DR

The paper tackles catastrophic forgetting in image restoration by introducing Continual Memory in Convolution (CMC), a modular kernel-generation scheme that splits convolution into a task-dependent vector and a shared memory bank, enabling sequential learning without altering backbone architectures. Kernel generation follows $K_w^{1\times m} = T_w^{1\times t} \cdot (M_w^{t\times m} \odot \mathscr{H})$, and the output uses $F_{out} = F_{in} \circledast K_w$, with memory capacity $t$ expandable as tasks arrive. The approach demonstrates that previous restoration knowledge can be reused to boost new task performance while keeping computational overhead low, outperforming several continual-learning baselines and maintaining performance across deraining, denoising, deblocking, and deblurring. The knowledge-base can be extended by increasing memory capacity, and the method is shown to be architecture-agnostic, scalable to similar and diverse degradation types, and practical for edge devices. This work provides a first-of-its-kind continual-learning framework for completely different image restoration tasks, offering a scalable, low-overhead path to lifelong restoration performance improvements.

Abstract

Continual learning is an emerging topic in the field of deep learning, where a model is expected to learn continuously for new upcoming tasks without forgetting previous experiences. This field has witnessed numerous advancements, but few works have been attempted in the direction of image restoration. Handling large image sizes and the divergent nature of various degradation poses a unique challenge in the restoration domain. However, existing works require heavily engineered architectural modifications for new task adaptation, resulting in significant computational overhead. Regularization-based methods are unsuitable for restoration, as different restoration challenges require different kinds of feature processing. In this direction, we propose a simple modification of the convolution layer to adapt the knowledge from previous restoration tasks without touching the main backbone architecture. Therefore, it can be seamlessly applied to any deep architecture without any structural modifications. Unlike other approaches, we demonstrate that our model can increase the number of trainable parameters without significantly increasing computational overhead or inference time. Experimental validation demonstrates that new restoration tasks can be introduced without compromising the performance of existing tasks. We also show that performance on new restoration tasks improves by adapting the knowledge from the knowledge base created by previous restoration tasks. The code is available at https://github.com/aupendu/continual-restore.

Figures (6)

• Figure 1: The proposed convolutional filter estimation block. $M_w$ is continual memory, $T_w$ is restoration task-specific weight vector, and $K_w$ is the estimated convolutional kernel.
• Figure 2: CMC layer during the first restoration task.
• Figure 3: Operations in CMC layer for $n^{th}$ restoration task.
• Figure 4: PSNR in dB vs Each epoch of training
• Figure 5: Qualitative evaluation of the effect of the knowledge sharing in our continual learning framework. 'Image 1' represents the outputs of the model without any knowledge sharing. 'Image 2' depicts the results of those models where the respective tasks are trained at last using the knowledge of all the previous tasks. (Zoom for the best view.)