Color Shift Estimation-and-Correction for Image Enhancement

TL;DR

The paper tackles color distortion in images with concurrent over- and under-exposure by leveraging pseudo-normal exposure features to estimate and correct color shifts. It introduces COSE, a color-space deformable convolution-based color-shift estimator, and COMO, a cross-attention-based color modulation module, integrated into a UNet-derived framework that jointly adjusts brightness and color. A pseudo-normal feature generator provides reference guidance, and a dual-loss scheme supervises both pseudo-normal construction and final output quality. Experiments on LCDP and MSEC show state-of-the-art performance with a lightweight model, offering improved color fidelity and visual quality in challenging mixed-exposure scenes.

Abstract

Images captured under sub-optimal illumination conditions may contain both over- and under-exposures. Current approaches mainly focus on adjusting image brightness, which may exacerbate the color tone distortion in under-exposed areas and fail to restore accurate colors in over-exposed regions. We observe that over- and under-exposed regions display opposite color tone distribution shifts with respect to each other, which may not be easily normalized in joint modeling as they usually do not have ``normal-exposed'' regions/pixels as reference. In this paper, we propose a novel method to enhance images with both over- and under-exposures by learning to estimate and correct such color shifts. Specifically, we first derive the color feature maps of the brightened and darkened versions of the input image via a UNet-based network, followed by a pseudo-normal feature generator to produce pseudo-normal color feature maps. We then propose a novel COlor Shift Estimation (COSE) module to estimate the color shifts between the derived brightened (or darkened) color feature maps and the pseudo-normal color feature maps. The COSE module corrects the estimated color shifts of the over- and under-exposed regions separately. We further propose a novel COlor MOdulation (COMO) module to modulate the separately corrected colors in the over- and under-exposed regions to produce the enhanced image. Comprehensive experiments show that our method outperforms existing approaches. Project webpage: https://github.com/yiyulics/CSEC.

Figures (7)

• Figure 1: Given images with both over- and under-exposures (a), SOTA methods Afifi_2021_CVPR_MSECWang_2022_ECCV_lcdp may still fail to correct color distortion (b,c). We show PCA results of pixels randomly sampled from MSEC Afifi_2021_CVPR_MSEC (images with either over- or under-exposed) (f) and LCDP Wang_2022_ECCV_lcdp (images with both over- and under-exposures) (g) datasets, and make two observations. First, we observe that under-exposed pixels (greenish dots) tend to have a reverse distribution shift compared to over-exposed pixels (reddish dots) in both datasets. Second, unlike MSEC Afifi_2021_CVPR_MSEC that contains normal-exposed pixels (bluish dots) in their 0 EV input images, images with both over- and under-exposures do not have such "reference pixels" as guidance. These two observations inspire us to estimate and correct such color shifts conditioned on the created pseudo-normal exposed features. Our method can properly adjust images with both over- and under-exposures (d).
• Figure 2: Overview of our proposed model. We first generate darkened features $F_D$ and brightened features $F_B$ using the UNet-based feature extractor. We then derive a pseudo-normal feature map $F_N$ using the generated brightened/darkened feature maps and the input image $I_x$. We further estimate the color shifts between the brightened/darkened color features $F_B$/$F_D$ and the created pseudo-normal feature map $F_N$ using the proposed Color Shift Estimation (COSE) module to obtain two individual offset maps $O_B$ and $O_D$. Finally, we modulate the image brightness and colors using the proposed Color Modulation (COMO) module, to produce the final output image.
• Figure 3: The proposed Color Shift Estimation (COSE) module aims to model the color distribution shifts in over- and under-exposed regions. We use three different convolutions to estimate the positional offsets, color shift offsets, and modulation scalars. We then utilize the color space deformable convolution in Eq. \ref{['eq:cdc']} to compute the color offsets and produce the offset feature map for brightened and darkened features.
• Figure 4: The proposed Color Modulation (COMO) module aims to adjust the brightness and correct the color shifts of the input image $I_x$, guided by the color offsets from our COSE modules. The COMO module takes the image $I_x$ and brighten/darken offsets $O_B$/$O_D$ as input and produces the enhanced image via a tailored cross-attention mechanism.
• Figure 5: Visual comparison between our method and state-of-the-art methods on LCDP Wang_2022_ECCV_lcdp dataset, which has images with both over-exposure and under-exposure.