GN-FR:Generalizable Neural Radiance Fields for Flare Removal

TL;DR

GN-FR tackles flare removal by reframing it as a multi-view problem and extending Generalizable Neural Radiance Fields with Flare-occupancy Mask Generation (FMG), View Sampler (VS), and Point Sampler (PS). It uses an unsupervised mask-guided loss to train without ground-truth flare-free images and leverages neighboring views to fill in flare-obscured details. The approach introduces a 3D flare dataset (17 scenes, 782 images, 80 flare patterns) and a real-flare pattern collection, enabling effective flare detection and removal within a NeRF-based pipeline. Empirically, GN-FR achieves state-of-the-art flare removal and novel-view synthesis under flare across synthetic and real data, surpassing baselines like NeRF and GNT by significant margins (e.g., cross-scene PSNR of $26.18$ vs $21.9$–$22.4$). The work demonstrates the practicality of multi-view information for de-flaring and sets up a foundation for broader, unsupervised flare handling in neural rendering.

Abstract

Flare, an optical phenomenon resulting from unwanted scattering and reflections within a lens system, presents a significant challenge in imaging. The diverse patterns of flares, such as halos, streaks, color bleeding, and haze, complicate the flare removal process. Existing traditional and learning-based methods have exhibited limited efficacy due to their reliance on single-image approaches, where flare removal is highly ill-posed. We address this by framing flare removal as a multi-view image problem, taking advantage of the view-dependent nature of flare artifacts. This approach leverages information from neighboring views to recover details obscured by flare in individual images. Our proposed framework, GN-FR (Generalizable Neural Radiance Fields for Flare Removal), can render flare-free views from a sparse set of input images affected by lens flare and generalizes across different scenes in an unsupervised manner. GN-FR incorporates several modules within the Generalizable NeRF Transformer (GNT) framework: Flare-occupancy Mask Generation (FMG), View Sampler (VS), and Point Sampler (PS). To overcome the impracticality of capturing both flare-corrupted and flare-free data, we introduce a masking loss function that utilizes mask information in an unsupervised setting. Additionally, we present a 3D multi-view flare dataset, comprising 17 real flare scenes with 782 images, 80 real flare patterns, and their corresponding annotated flare-occupancy masks. To our knowledge, this is the first work to address flare removal within a Neural Radiance Fields (NeRF) framework.

Figures (8)

• Figure 1: Our method: GN-FR is built on significant modifications to the GNT framework with flare masking loss, an effective view sampler and a Point Sampler. Our method consistently removes flare across the views on synthetic flare-imposed scenes and real scenes with flare.
• Figure 2: Overview of GN-FR: 1) FMG: Flare-occupancy Mask Generation network: It generates a binary mask for each input view to detect the regions w/ and w/o flare. 2) View Sampler: Samples the views minimally affected by flare using the flare occupancy mask as a cue. 3) Point Sampler: An attention masking-based sampling mechanism in the view transformer to guide the network for better flare removal.
• Figure 5: Qualitative results on Synthetic dataset: GN-FR framework is able to consistently remove the diverse flares present in the synthetically imposed flare images. As the insets highlight, GN-FR surpasses other methods in removing flare artifacts.
• Figure 6: Qualitative results on Real dataset: Our GN-FR framework is able to remove the flare artifacts from flare-corrupted images effectively. It is able to render more visually appealing results consistently compared to other methods.
• Figure 7: Flare removal results on captured Real dataset: GN-FR is able to consistently remove the flare-artifacts on different scenes of our real flare-3D dataset and also reproduce the lost scene information covered up due to flare effects(Zoom-in for better visualization).