ECMamba: Consolidating Selective State Space Model with Retinex Guidance for Efficient Multiple Exposure Correction

TL;DR

A novel framework based on Mamba for Exposure Correction (ECMamba) with dual pathways, each dedicated to the restoration of reflectance and illumination map, respectively is introduced, inspired by Mamba which demonstrates powerful and highly efficient sequence modeling.

Abstract

Exposure Correction (EC) aims to recover proper exposure conditions for images captured under over-exposure or under-exposure scenarios. While existing deep learning models have shown promising results, few have fully embedded Retinex theory into their architecture, highlighting a gap in current methodologies. Additionally, the balance between high performance and efficiency remains an under-explored problem for exposure correction task. Inspired by Mamba which demonstrates powerful and highly efficient sequence modeling, we introduce a novel framework based on Mamba for Exposure Correction (ECMamba) with dual pathways, each dedicated to the restoration of reflectance and illumination map, respectively. Specifically, we firstly derive the Retinex theory and we train a Retinex estimator capable of mapping inputs into two intermediary spaces, each approximating the target reflectance and illumination map, respectively. This setup facilitates the refined restoration process of the subsequent Exposure Correction Mamba Module (ECMM). Moreover, we develop a novel 2D Selective State-space layer guided by Retinex information (Retinex-SS2D) as the core operator of ECMM. This architecture incorporates an innovative 2D scanning strategy based on deformable feature aggregation, thereby enhancing both efficiency and effectiveness. Extensive experiment results and comprehensive ablation studies demonstrate the outstanding performance and the importance of each component of our proposed ECMamba. Code is available at https://github.com/LowlevelAI/ECMamba.

Figures (5)

• Figure 1: (a) T-SNE t-SNE visualization of distributions of modulated reflectance ($\mathbf{R}^{\prime}$), restored reflectance ($\mathbf{R}_{out}$) and the final output ($\mathbf{I}_{out}$) for Under-Exposed (UE) and Over-Exposed (OE) images. Compared to the input data, modulated reflectance ($\mathbf{R}^{\prime}$) demonstrates closer approximation of Normal-Exposed (NE) images. Besides, compared to the restored reflectance ($\mathbf{R}_{out}$), our final output ($\mathbf{I}_{out}$) are better aligned with NE data. (b) Visual result of $\mathbf{R}^{\prime}$, $\mathbf{R}_{out}$ and $\mathbf{I}_{out}$ produced by our method. From column $2$-$4$, we observe a noticeable improvement on color preservation and structure recovery, which demonstrates the importance of our introduced two-branch Retinex-based pipeline and the effectiveness of our proposed ECMamba network.
• Figure 2: The overall architecture of our proposed Retinex-based framework for exposure correction, which includes a Retinex estimator $\mathcal{E}$ and primary restoration network $\mathcal{M}_{R}$ and $\mathcal{M}_{L}$.
• Figure 3: The details of our proposed Retinex-SS2D layer. We firstly fuse the input feature $\mathbf{F}_{in}$ and the Retinex guidance $\mathbf{F}_{in}$. Then we propose an innovative Feature-Aware 2D Selective State-spce Mechanism, which utilizes Deformable Convolution (DCN) for feature aggregation. Then we propose the feature-aware scanning strategy based on the activation response map derived from DCN. Compared to other 2D scanning methods, our approach generates a sequence ordered by feature importance, thereby maximizing the robust sequence modeling capabilities of Mamba.
• Figure 4: Visual comparison results on ME dataset. Compared to other exposure correction methods, our ECMamba excels in color preservation and structure recovery.
• Figure 5: Visual comparisons between ECMamba and other methods on SICE dataset. Our proposed ECMamba achieves compelling visual performance both on over-exposed and under-exposed images.