Efficient Diffusion as Low Light Enhancer

TL;DR

Efficient Diffusion as Low Light Enhancer tackles the inefficiency of diffusion-based LLIE by identifying fitting errors and inference gaps as key performance bottlenecks. It introduces Reflectance-Aware Trajectory Refinement and a distillation framework ReDDiT to transfer a multi-step teacher trajectory to a lightweight student, achieving strong restoration with only 2 steps and state-of-the-art results at 4 and 8 steps across 10 benchmarks. The method combines trajectory distillation, a reflectance-based residual prior, and auxiliary pixel and perceptual losses, yielding robust improvements in PSNR, SSIM, LPIPS, and NIQE while offering practical speedups. Across diverse datasets including LOLv1/2, SID, SDSD, and unpaired collections, ReDDiT demonstrates both high-quality restoration and strong efficiency, moving diffusion-based LLIE closer to real-time applicability on edge devices.

Abstract

The computational burden of the iterative sampling process remains a major challenge in diffusion-based Low-Light Image Enhancement (LLIE). Current acceleration methods, whether training-based or training-free, often lead to significant performance degradation, highlighting the trade-off between performance and efficiency. In this paper, we identify two primary factors contributing to performance degradation: fitting errors and the inference gap. Our key insight is that fitting errors can be mitigated by linearly extrapolating the incorrect score functions, while the inference gap can be reduced by shifting the Gaussian flow to a reflectance-aware residual space. Based on the above insights, we design Reflectance-Aware Trajectory Refinement (RATR) module, a simple yet effective module to refine the teacher trajectory using the reflectance component of images. Following this, we introduce \textbf{Re}flectance-aware \textbf{D}iffusion with \textbf{Di}stilled \textbf{T}rajectory (\textbf{ReDDiT}), an efficient and flexible distillation framework tailored for LLIE. Our framework achieves comparable performance to previous diffusion-based methods with redundant steps in just 2 steps while establishing new state-of-the-art (SOTA) results with 8 or 4 steps. Comprehensive experimental evaluations on 10 benchmark datasets validate the effectiveness of our method, consistently outperforming existing SOTA methods.

Figures (10)

• Figure 1: ReDDiT shifts the teacher trajectory from the original Gaussian distribution to a residual space, effectively reducing the sampling gap. Subsequently, it refines the teacher trajectory towards the ground truth trajectory to mitigate fitting errors.
• Figure 2: Pipiline of our proposed ReDDiT. The distillation process involves two parts: teacher model leverages the estimated reflectance to refine its trajectory and student model's trajectory is guided by teacher's trajectory, via a distillation loss. TD denotes the Trajectory Decoder while RATR denotes the Reflectance-Aware Trajectory Refinement.
• Figure 3: Qualitative results on LOLv1 (top) , LOLv2-real (middle), and LOLv2-synthetic (bottom). Patches highlighted in each image by the red box indicate that ReDDiT effectively enhances the visibility, preserves the color and generates finer details in normal light images. Zoom in to better observe the image details.
• Figure 4: Qualitative results on NPE, VV, LIME, DICM, and MEF. Compared to Retinexformer, our method effectively enhances the visibility and preserves the color in normal light images. Zoom in to better observe the image details.
• Figure 5: Quantitative comparisons on the LoLv1 and LoLv2 (Real and Synthetic) datasets with other accelerate methods. Popular acceleration methods experience a significant decline in performance as the number of steps decreases. Our method outperforms other acceleration techniques as well as the original teacher model (in terms of DDIM performance).

Theorems & Definitions (1)

• Corollary 1: Proof in the supplementary material