DeepSPG: Exploring Deep Semantic Prior Guidance for Low-light Image Enhancement with Multimodal Learning

TL;DR

DeepSPG targets semantic-aware low-light image enhancement by combining Retinex-based reflectance refinement with multimodal semantic priors. It fuses image-level semantics from a frozen segmentation model and text-level semantics via CLIP, operationalized through a Semantic-aware Embedding Module and a CLIP-based semantic constraint, with a coarse-to-fine multi-scale structure. The method uses pixel, edge, semantic, and multimodal losses to enforce visual fidelity, structural integrity, and semantic coherence, achieving state-of-the-art results on five LLIE benchmarks with strong efficiency. This work demonstrates that explicit semantic guidance can stabilize restoration in extremely dark regions and suggests a practical path toward semantically consistent LLIE in real-world applications.

Abstract

There has long been a belief that high-level semantics learning can benefit various downstream computer vision tasks. However, in the low-light image enhancement (LLIE) community, existing methods learn a brutal mapping between low-light and normal-light domains without considering the semantic information of different regions, especially in those extremely dark regions that suffer from severe information loss. To address this issue, we propose a new deep semantic prior-guided framework (DeepSPG) based on Retinex image decomposition for LLIE to explore informative semantic knowledge via a pre-trained semantic segmentation model and multimodal learning. Notably, we incorporate both image-level semantic prior and text-level semantic prior and thus formulate a multimodal learning framework with combinatorial deep semantic prior guidance for LLIE. Specifically, we incorporate semantic knowledge to guide the enhancement process via three designs: an image-level semantic prior guidance by leveraging hierarchical semantic features from a pre-trained semantic segmentation model; a text-level semantic prior guidance by integrating natural language semantic constraints via a pre-trained vision-language model; a multi-scale semantic-aware structure that facilitates effective semantic feature incorporation. Eventually, our proposed DeepSPG demonstrates superior performance compared to state-of-the-art methods across five benchmark datasets. The implementation details and code are publicly available at https://github.com/Wenyuzhy/DeepSPG.

Figures (6)

• Figure 1: Comparisons between our DeepSPG (bottom) and previous methods (SNR-Net snr_net, top). After enhancement, significant color distortions and noise can be observed in SNR-Net, while DeepSPG can recover high-quality results.
• Figure 2: Overall framework of DeepSPG. (a) The main coarse-to-fine enhancement pipeline incorporates both image-level and text-level semantic priors. (b) Multi-modal based semantic supervision leveraging both images and text, where semantic consistency is enforced by computing the cosine similarity between textual prompts and extracted features. (c) The Semantic-aware Embedding Module (SEM) dynamically aligns semantic and reflectance branch features.
• Figure 3: Visual results on LOL-v1 retinex (top) and SID smid (bottom). Brightness correction is equally applied to all cropped patches (blue box and pink box) for better detail comparison. Previous methods often fail due to noise, color distortion, or producing blurry and under- or over-exposed images. In contrast, our DeepSPG effectively removes noise and reconstructs well-exposed image details. (Please Zoom in for the best view.)
• Figure 4: Visual comparisons of Restormer restormer, MIRNet mirnet, SNR-Net snr_net, Retinexformer cai2023retinexformer, HAIR cao2024hair, and our DeepSPG on some extremely low-light and noisy scenes of the SID smid dataset. Previous methods often fail due to noise, color distortion, or producing blurry and under- or over-exposed images. In contrast, our PiCat effectively removes noise and reconstructs well-exposed image details. (Please Zoom in for the best view.)
• Figure 5: Visual results of break-down ablations on the LOL-v2-real dataset lol_v2. As can be seen, without both image-level and text-level semantic prior guidance, texture blur and halation appear in the middle and right plotted patches next to the corresponding result, respectively.