All-in-One Image Restoration via Causal-Deconfounding Wavelet-Disentangled Prompt Network

TL;DR

Extensive experiments on two all-in-one settings prove the effectiveness and superior performance of the proposed Causal-deconfounding Wavelet-disentangled Prompt Network (CWP-Net) to perform effective AiOIR.

Abstract

Image restoration represents a promising approach for addressing the inherent defects of image content distortion. Standard image restoration approaches suffer from high storage cost and the requirement towards the known degradation pattern, including type and degree, which can barely be satisfied in dynamic practical scenarios. In contrast, all-in-one image restoration (AiOIR) eliminates multiple degradations within a unified model to circumvent the aforementioned issues. However, according to our causal analysis, we disclose that two significant defects still exacerbate the effectiveness and generalization of AiOIR models: 1) the spurious correlation between non-degradation semantic features and degradation patterns; 2) the biased estimation of degradation patterns. To obtain the true causation between degraded images and restored images, we propose Causal-deconfounding Wavelet-disentangled Prompt Network (CWP-Net) to perform effective AiOIR. CWP-Net introduces two modules for decoupling, i.e., wavelet attention module of encoder and wavelet attention module of decoder. These modules explicitly disentangle the degradation and semantic features to tackle the issue of spurious correlation. To address the issue stemming from the biased estimation of degradation patterns, CWP-Net leverages a wavelet prompt block to generate the alternative variable for causal deconfounding. Extensive experiments on two all-in-one settings prove the effectiveness and superior performance of our proposed CWP-Net over the state-of-the-art AiOIR methods.

Figures (13)

• Figure 1: (a) The illustration of spurious correlation. We present the statistical analysis of four selected categories in the training data of five tasks. Detailed dataset information is provided in Sec. \ref{['sec:dataset']}. (b) The prediction accuracy of five degradation patterns on imbalanced test data and balanced test data. ResNet-34 he2016deep, VGG-16 simonyan2014very, and DA-CLIP luo2023controlling are utilized as degradation classifiers. The training and testing data used in conventional AiOIR models are considered imbalanced. Balanced means that the task data is not limited to the scenarios present in the training set, but rather it is rich and comprehensive. All classifiers are trained on imbalanced training data. The balanced test data is constructed with details outlined in Sec. \ref{['sec:exp_unbiased']}.
• Figure 2: The SCM graph of AiOIR. (a) A unified causal framework for existing ideal AiOIR methods. (b) Two challenges of practical AiOIR via causal lens. The dashed line $\dashleftarrow \dashrightarrow$ represents the spurious correlation, and the grey node denotes the unobservability of the variable. (c) Causal deconfounding through alternative variable $P$. The red path and the symbol "$\textcolor{red}{\times}$" indicate the use of the wavelet attention module to eliminate the spurious correlation between $C$ and $T$, allowing the restoration network to focus on degraded regions rather than semantic features.
• Figure 3: The pipeline of our causal analysis and the corresponding causality-guided method, i.e., CWP-Net.
• Figure 4: (a) The distortion severity of five degradation patterns on each wavelet subband. The "PSNR" is calculated with a higher value indicating milder distortion. Here, we roughly consider items with a PSNR above 30 dB (green dashed line) as distortion-mild subbands, and those below 30 dB as distortion-severe subbands. (b) Experiment results of introducing prompt components to distortion-mild (e.g., "HH" and "HL" for derain task) and distortion-severe (e.g., "LH" and "LL" for derain task) subbands of features in AiOIR model.
• Figure 5: Architecture of our proposed CWP-Net. (a) provides an overview of the U-shaped network for AiOIR, in which WAE and WAD are symmetrically placed at each level of the encoder and decoder, and WPB is inserted into the skip connection. (b) WAE and WAD extract attention maps of different wavelet subbands in both spatial and channel dimensions. (c) WPB generates weighted prompted wavelet subbands for backdoor adjustment based on the low-frequency wavelet attention map. $Z_{*}$ and $P_{*}$ denote the wavelet subbands before and after prompt-based modulation, respectively.

Theorems & Definitions (5)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Theorem 1: Rules of do Calculus