Super-resolving Real-world Image Illumination Enhancement: A New Dataset and A Conditional Diffusion Model

TL;DR

Comprehensive quantitative and qualitative experimental results on the real-world benchmark datasets demonstrate the feasibility and effectivenesses of the proposed conditional diffusion probabilistic model on Raw sensor data.

Abstract

Most existing super-resolution methods and datasets have been developed to improve the image quality in well-lighted conditions. However, these methods do not work well in real-world low-light conditions as the images captured in such conditions lose most important information and contain significant unknown noises. To solve this problem, we propose a SRRIIE dataset with an efficient conditional diffusion probabilistic models-based method. The proposed dataset contains 4800 paired low-high quality images. To ensure that the dataset are able to model the real-world image degradation in low-illumination environments, we capture images using an ILDC camera and an optical zoom lens with exposure levels ranging from -6 EV to 0 EV and ISO levels ranging from 50 to 12800. We comprehensively evaluate with various reconstruction and perceptual metrics and demonstrate the practicabilities of the SRRIIE dataset for deep learning-based methods. We show that most existing methods are less effective in preserving the structures and sharpness of restored images from complicated noises. To overcome this problem, we revise the condition for Raw sensor data and propose a novel time-melding condition for diffusion probabilistic model. Comprehensive quantitative and qualitative experimental results on the real-world benchmark datasets demonstrate the feasibility and effectivenesses of the proposed conditional diffusion probabilistic model on Raw sensor data. Code and dataset will be available at https://github.com/Yaofang-Liu/Super-Resolving

Figures (13)

• Figure 1: Visual comparisons ($\times 4$ SR) on the SRRIIE dataset for super-resolving real-world image illumination enhancement tasks. The low-quality image is captured with the camera settings of -5 EV and ISO 4000. The complicated noises make it difficult to restore truthful image information. Our conditional DPM-based method using Raw sensor data generates better structural details.
• Figure 2: Our data collection and processing pipeline for one scene. An ILDC with an optical zoom lens is used to collect real Raw sensor data with a wide range of EV and ISO levels. Camera ISP is employed to convert Raw images to sRGB images and only the centered croppings are retained. Each low-light image sequence contain 9 low-light Raw/sRGB images with complicated real-world noises and low intensities. While each high-resolution image sequence contains 3 noise-free normal-light high-resolution images as the ground truth. These paired images model the real-world degradation process for for super-resolving image illumination enhancement (SRRIIE) tasks.
• Figure 3: Simple cases to illustrate the motivation. (b)-(e) are the corresponding cropped and zoomed-in regions as denoted by the red box in (a). The synthetic Gaussian-Poisson noisy image (b) or the image captured with the low ISO level (c) deviates largely from real-world noisy images (d) - (e) in terms of visual comparison and statistical image gradient distributions (f).
• Figure 4: An overview of our conditional DPM with the forward diffusion (dashed line) and reverse process (solid line). Our method generates photo-realistic image structural details conditioned on Raw sensor data $\pi{(\widetilde{x})}$ (Eq. (\ref{['eq: DPM7']})). The proposed time-melding condition (Eq. (\ref{['eq: DPM7']})) improves the reverse generation process (Eq. (\ref{['eq: DPM8']})) by capturing temporal coherence and consistency across relevant time points.
• Figure 5: Qualitative visual comparisons for $\times$2 SR (-3 EV, ISO 3200) on our SRRIIE dataset. The compared methods restore noticeable color distortions and artifacts while our method generates photo-realistic results with nature face structures.