Difflare: Removing Image Lens Flare with Latent Diffusion Model

TL;DR

Difflare tackles lens flare removal, a challenging local image degradation, by leveraging pre-trained latent diffusion priors through latent-space fine-tuning. It introduces a Structural Guidance Injection Module (SGIM) to inject multi-scale structural cues and an Adaptive Feature Fusion Module (AFFM) guided by a Luminance Gradient Prior (LGP) to preserve flare-free regions. The method achieves strong fidelity and perceptual quality on real-world flare data, outperforming prior approaches while reducing training costs by operating in latent space. This approach demonstrates the value of latent-diffusion priors for local degradation tasks and suggests broader applicability to related image restoration problems.

Abstract

The recovery of high-quality images from images corrupted by lens flare presents a significant challenge in low-level vision. Contemporary deep learning methods frequently entail training a lens flare removing model from scratch. However, these methods, despite their noticeable success, fail to utilize the generative prior learned by pre-trained models, resulting in unsatisfactory performance in lens flare removal. Furthermore, there are only few works considering the physical priors relevant to flare removal. To address these issues, we introduce Difflare, a novel approach designed for lens flare removal. To leverage the generative prior learned by Pre-Trained Diffusion Models (PTDM), we introduce a trainable Structural Guidance Injection Module (SGIM) aimed at guiding the restoration process with PTDM. Towards more efficient training, we employ Difflare in the latent space. To address information loss resulting from latent compression and the stochastic sampling process of PTDM, we introduce an Adaptive Feature Fusion Module (AFFM), which incorporates the Luminance Gradient Prior (LGP) of lens flare to dynamically regulate feature extraction. Extensive experiments demonstrate that our proposed Difflare achieves state-of-the-art performance in real-world lens flare removal, restoring images corrupted by flare with improved fidelity and perceptual quality. The codes will be released soon.

Figures (4)

• Figure 1: (a) Real-world captured RF and SF; (b) Diagram of the origin of lens flare. Ideally, rays from the point light source incident to the lens are intended to be focused at a single point on the camera sensor (gray rays). However, due to the dust, wear and scratches on the surfaces of the lenses and the reflection between the air-glass interface of the lens, the incident rays might be scattered or reflected to unexpected locations, leading to unwilling artifacts on the captured image.
• Figure 2: (a) Overview of our proposed Difflare. (b) We first utilize the Structural Guidance Injection Module (SGIM) to finetune the frozen Pre-Trained Diffusion Model (PTDM). The multi-scale features extracted by the SGIM is transformed to the corresponding resolution layer of PTDM through Spatially-Adaptive Normalization (SPADE) park2019SPADE layers. (c) Additionally, motivated by StableSR wang2023exploiting, we introduce a Adaptive Feature Fusion Module (AFFM) to maintain the fidelity between input image and restored image. The AFFM accepts the feature from VQ-GAN encoder and VQ-GAN decoder, and outputs a fusion of both features. The whole process is guided by the Luminance Gradient Prior (LGP) mask via a modification of the self-attention map.
• Figure 3: Visual Comparison on Flare7K dai2022flare7k testset. Our proposed method can effectively remove lens flare and unwilling artifacts, while harmonizing the recovered light source and the background.
• Figure 4: Visual comparison on the effect of AFFM. Without AFFM, SGIM can effectively remove lens flare, but there are significant distortions on flare-free areas. When AFFM is employed, the fidelity of flare-free areas has been maintained between the input image and the restored image.