Boosting Fidelity for Pre-Trained-Diffusion-Based Low-Light Image Enhancement via Condition Refinement

TL;DR

The paper addresses fidelity gaps in Pre-Trained-Diffusion-Based low-light image enhancement by identifying deficiencies in conditional latent modeling and the lack of bidirectional interaction with the noisy latent. It proposes a two-stage latent refinement method guided by a generative prior and a bidirectional interaction mechanism that allows refined and noisy latents to influence each other during diffusion, designed as a plug-and-play enhancement for existing PTDB systems. Through multi-scale pyramidal priors, visual encoders, and attention-guided regression, the approach achieves substantial fidelity gains while preserving the qualitative advantages of diffusion-based restoration, demonstrated across LOL, SID, and SMID datasets and downstream tasks. The method also reports improved no-reference image quality metrics and shows generalization to related restoration problems, albeit with additional computational costs. Overall, the work offers a practical pathway to boost fidelity in PTDB LLIE and related tasks by refining conditioning latents and enabling bidirectional latent exchange.

Abstract

Diffusion-based methods, leveraging pre-trained large models like Stable Diffusion via ControlNet, have achieved remarkable performance in several low-level vision tasks. However, Pre-Trained Diffusion-Based (PTDB) methods often sacrifice content fidelity to attain higher perceptual realism. This issue is exacerbated in low-light scenarios, where severely degraded information caused by the darkness limits effective control. We identify two primary causes of fidelity loss: the absence of suitable conditional latent modeling and the lack of bidirectional interaction between the conditional latent and noisy latent in the diffusion process. To address this, we propose a novel optimization strategy for conditioning in pre-trained diffusion models, enhancing fidelity while preserving realism and aesthetics. Our method introduces a mechanism to recover spatial details lost during VAE encoding, i.e., a latent refinement pipeline incorporating generative priors. Additionally, the refined latent condition interacts dynamically with the noisy latent, leading to improved restoration performance. Our approach is plug-and-play, seamlessly integrating into existing diffusion networks to provide more effective control. Extensive experiments demonstrate significant fidelity improvements in PTDB methods.

Figures (9)

• Figure 1: (a) Traditional restoration-only methods (SNR-aware network xu2022snr in the top and Diff-L jiang2023low that is trained from scratch) achieve high fidelity, while users may not accept these noisy results in practice. They require improvements in detail and aesthetic quality. (b) In contrast, PTDB methods (e.g., DiffBIR lin2024diffbir here) leverage pre-trained diffusion models to generate detailed, sharp, and clean images, but often at the loss of fidelity (distortion). Fidelity is critical for some regions, e.g., letters. (c) Unlike prior works that focus on enhancing control structures for diffusion models, our approach directly refines the conditional latents and their interaction with noisy latents at various steps. This allows us to preserve advantages of pre-trained diffusion models while improving fidelity. Our method is plug-and-play for all PTDB approaches.
• Figure 2: VAE reconstructions may contain distortions, meaning the conditional latent cannot fully capture the entire information of input image. Conditional latents should be refined.
• Figure 3: Illustration of our strategy for conditional latent refinement and interaction. (a) The input image and its enhanced version are used for refinement through information-lossless operations, such as space-to-depth and a visual spatial encoder. (b) These inputs, $t_i$ and $t_l$, feed into the latent refinement procedure, which includes the generative prior and final latent prediction, yielding $\hat{h}_{m,0}$ and $\hat{l}_m$. (c) The refined latent $\hat{l}_m$ then interacts bidirectionally with the noisy latent $x_t$ in the diffusion backbone $F$, promoting residuals $\Delta_{x_t}$ and $\Delta_{m_t}$, thereby enhancing the performance of $C$. "$D$ times iteration" refers to $D$ diffusion steps.
• Figure 4: Visual samples to show the effects of our latent refinement strategy. We decode the generative prior $\hat{\boldsymbol{h}}_{m,0}$ (at its highest scale, i.e., the first four channels), the refined latent $\hat{\boldsymbol{l}}_m$, and the ground truth latent. The results show that the prior indicates the correct improvement direction, and the refined latent is closer to the ground truth.
• Figure 5: Visual comparisons on different datasets with various network structures on LOL-real.