RCNet: Deep Recurrent Collaborative Network for Multi-View Low-Light Image Enhancement

TL;DR

A multi-view enhancement framework based on the Recurrent Collaborative Network (RCNet), where the recurrent feature enhancement, alignment, and fusion (ReEAF) module, where intra-view feature enhancement (Intra-view EN) followed by inter-view feature alignment and fusion (Inter-view AF) is performed to model intra-view and inter-view feature propagation via multi-view collaboration.

Abstract

Scene observation from multiple perspectives would bring a more comprehensive visual experience. However, in the context of acquiring multiple views in the dark, the highly correlated views are seriously alienated, making it challenging to improve scene understanding with auxiliary views. Recent single image-based enhancement methods may not be able to provide consistently desirable restoration performance for all views due to the ignorance of potential feature correspondence among different views. To alleviate this issue, we make the first attempt to investigate multi-view low-light image enhancement. First, we construct a new dataset called Multi-View Low-light Triplets (MVLT), including 1,860 pairs of triple images with large illumination ranges and wide noise distribution. Each triplet is equipped with three different viewpoints towards the same scene. Second, we propose a deep multi-view enhancement framework based on the Recurrent Collaborative Network (RCNet). Specifically, in order to benefit from similar texture correspondence across different views, we design the recurrent feature enhancement, alignment and fusion (ReEAF) module, in which intra-view feature enhancement (Intra-view EN) followed by inter-view feature alignment and fusion (Inter-view AF) is performed to model the intra-view and inter-view feature propagation sequentially via multi-view collaboration. In addition, two different modules from enhancement to alignment (E2A) and from alignment to enhancement (A2E) are developed to enable the interactions between Intra-view EN and Inter-view AF, which explicitly utilize attentive feature weighting and sampling for enhancement and alignment, respectively. Experimental results demonstrate that our RCNet significantly outperforms other state-of-the-art methods. All of our dataset, code, and model will be available at https://github.com/hluo29/RCNet.

Figures (10)

• Figure 1: Illustration of multi-view low-light images and the enhanced results of state-of-the-art methods. (a)$\sim$(c): three different views in the same scene, with each composed of low-light image and bright result corrected by Gamma transformation. (d)$\sim$(h): the results of SRIEfu2016weighted, BIMEFying2017bio, RetinexNetwei2018deep, DRBNyang2020fidelity and our RCNet, using the low-light View#2 as input. (i): the normal-light version of low-light View#2.
• Figure 2: Illustration of our MVLT dataset construction and statistics: (a) we adopt the DISTS metric to compute the similarity score with the threshold T for multi-view triplets selection; (b) the example triplets of normal-light images; (c) the differential low-light synthesis is composed of brightness reduction and noise simulation; (d) the intensity distribution of low/normal-light images in training and testing set, respectively. Please zoom in for a better visualization.
• Figure 3: Illustration of our proposed multi-view low-light enhancement framework: (i) the multi-view low-light images are grouped into a triplet $\mathcal{D}$ including a primary view ($x_{2}$) and two auxiliary views ($x_{1}$ and $x_{3}$); (ii) a shared encoder is utilized as the multi-scale feature extractor to obtain multi-view features in different scales from three low-light input views; (iii) the recurrent feature enhancement, alignment and fusion (ReEAF) module is embedded to integrate primary view features via multi-view collaboration. In each recurrent unit, the ReEAF is composed of Intra-view Enhancement (Intra-view EN) followed by Inter-view Alignment and Fusion (Inter-view AF); and (iv) the finally enhanced image $\mathcal{R}_{2}$ corresponding to primary view$x_{2}$ could be produced by a single convolutional layer at the end of the fusion stage.
• Figure 4: Illustration of the recurrent feature enhancement-alignment-fusion (ReEAF) module.
• Figure 5: Illustration of the feature alignment by searching top K similar patches in a local region.