Explore Internal and External Similarity for Single Image Deraining with Graph Neural Networks

TL;DR

This work tackles single-image deraining by exploiting non-local patch recurrence through a graph-based framework called MSGNN. It jointly models internal non-locality across multi-scale rainy inputs and external non-locality with an exemplar using a kNN graph and attentional aggregation within an end-to-end network. MSGNN demonstrates state-of-the-art performance on five synthetic datasets and a real-world dataset, while using fewer parameters than some baselines. The approach promises robust rain removal with preserved details and offers code for replication.

Abstract

Patch-level non-local self-similarity is an important property of natural images. However, most existing methods do not consider this property into neural networks for image deraining, thus affecting recovery performance. Motivated by this property, we find that there exists significant patch recurrence property of a rainy image, that is, similar patches tend to recur many times in one image and its multi-scale images and external images. To better model this property for image detaining, we develop a multi-scale graph network with exemplars, called MSGNN, that contains two branches: 1) internal data-based supervised branch is used to model the internal relations of similar patches from the rainy image itself and its multi-scale images and 2) external data-participated unsupervised branch is used to model the external relations of the similar patches in the rainy image and exemplar. Specifically, we construct a graph model by searching the k-nearest neighboring patches from both the rainy images in a multi-scale framework and the exemplar. After obtaining the corresponding k neighboring patches from the multi-scale images and exemplar, we build a graph and aggregate them in an attentional manner so that the graph can provide more information from similar patches for image deraining. We embed the proposed graph in a deep neural network and train it in an end-to-end manner. Extensive experiments demonstrate that the proposed algorithm performs favorably against eight state-of-the-art methods on five public synthetic datasets and one real-world dataset. The source codes will be available at https://github.com/supersupercong/MSGNN.

Figures (9)

• Figure 1: Illustration of our main idea. The proposed method consists of two branches: Supervised Branch and Unsupervised Branch. The Supervised Branch reasons the internal patch correlation on the multi-scale images, while the unsupervised branch utilizes the learning ability of graph neural networks to explore the external patch correlation with unseen images as exemplars to learn more rainy conditions for better rain removal.
• Figure 2: Illustration of the internal similarity in multi-scale images and external similarity in the exemplar. As can be seen, patch recurrence property indeed exists in internal multi-scale images and external exemplars.
• Figure 3: Diagram of the proposed MSGNN. Our method consists of two branches: an internal data-based supervised branch and an external data-participated unsupervised branch. In the supervised branch, we explore the internal property of multi-scale images by reasoning the multi-scale patch correlations to help the original-scale image learn more internal image information. In the unsupervised branch, we build the external patch correlations between input images and another external image as an exemplar by utilizing the superiority of the graph network which can search similar patches between any two samples so that the network is able to learn more rainy conditions for better rain removal. Graph Model is illustrated in Figure \ref{['fig: graph model']}.
• Figure 4: Illustration of the graph model which is divided into two steps: nearest neighbor search and attentional aggregation.
• Figure 5: Comparisons with state-of-the-art methods on synthetic dataset. Our proposed MSGNN is able to restore better texture.