LightsOut: Diffusion-based Outpainting for Enhanced Lens Flare Removal

TL;DR

LightsOut tackles the challenge of lens flare removal when off-frame light sources are incomplete by introducing a diffusion-based outpainting framework. The method combines a multitask regression module to predict off-frame light sources with a LoRA-finetuned diffusion inpainting model conditioned on those predictions, producing realistic outpainted scenes that provide better context for subsequent SIFR. It is designed as a plug-and-play preprocessing step that enhances existing SIFR methods without retraining them, and extensive experiments show consistent improvements across real and synthetic datasets, including challenging incomplete-light-source scenarios. The approach advances practical flare removal by explicitly modeling and reconstructing illumination context, improving both visual realism and downstream task performance, such as object detection.

Abstract

Lens flare significantly degrades image quality, impacting critical computer vision tasks like object detection and autonomous driving. Recent Single Image Flare Removal (SIFR) methods perform poorly when off-frame light sources are incomplete or absent. We propose LightsOut, a diffusion-based outpainting framework tailored to enhance SIFR by reconstructing off-frame light sources. Our method leverages a multitask regression module and LoRA fine-tuned diffusion model to ensure realistic and physically consistent outpainting results. Comprehensive experiments demonstrate LightsOut consistently boosts the performance of existing SIFR methods across challenging scenarios without additional retraining, serving as a universally applicable plug-and-play preprocessing solution. Project page: https://ray-1026.github.io/lightsout/

Figures (15)

• Figure 1: Motivation for outpainting incomplete off-frame light sources. (a) With complete off-frame light sources, state-of-the-art SIFR methods effectively remove lens flare artifacts. (b) In scenarios lacking complete views of off-frame sources, these methods degrade significantly, leaving noticeable artifacts. This highlights the importance of complete light source context, motivating our proposed outpainting solution.
• Figure 2: Overview of our proposed three-stage pipeline. (a) Light source prediction and conditioning: We introduce a multitask regression module to accurately predict off-frame or incomplete light source parameters (positions, radii, and confidences). These predicted parameters guide a rendering function to generate the corresponding light source mask. (b) Light source outpainting: Leveraging a LoRA fine-tuned diffusion-based inpainting model with light source conditioning, our approach accurately outpaints both missing off-frame light sources and associated flare artifacts, producing visually coherent and realistic results. (c) SIFR boosting: Our generated outpainted images serve as enhanced inputs to existing SIFR methods, significantly improving their performance on previously challenging scenarios with incomplete light source information. The proposed pipeline thus effectively operates as a plug-and-play module to boost existing flare removal models.
• Figure 3: Overview of the multitask regression module. Our model performs multitask regression to simultaneously predict two essential components: the physical parameters $\mathbf{P}$ and the corresponding confidence probabilities $\mathbf{c}$ for potential light sources. During training, these are supervised by a designed multitask loss. At inference, predicted parameters are integrated to generate light source masks $M_\text{L}$.
• Figure 4: Qualitative comparison of lens flare removal results. We compare state-of-the-art SIFR methods (Zhou et al.zhou2023improving, Flare7K++dai2023flare7k++, and MFDNet jiang2024mfdnet) alone and combined with our proposed method in two challenging scenarios: (top) no visible light sources and (bottom) incomplete light sources. Integrating our outpainting method ("Ours +") significantly improves flare removal quality, producing results closer to ground truth.
• Figure 5: Qualitative comparison of our outpainting results. We qualitatively compare our method with SD-Inpainting rombach2022high, SDXL-Inpainting podell2023sdxl, and PowerPaint zhuang2024task. Our method produces more realistic outpainting results, accurately capturing flare artifacts and aligning closely with real-world scenes.