Learning to See Through Dazzle

TL;DR

This work tackles laser dazzle and sensor saturation by integrating a wavefront-coded phase mask with a Sandwich Generative Adversarial Network (SGAN) that performs end-to-end restoration from phase-coded, laser-dazzled imagery. The SGAN wraps a learnable non-blind deconvolution between two GANs and leverages Fourier feature representations to reduce spectral bias, enabling robust recovery under varying laser strengths, angles, and noise. Three SGAN variants (SGAN-B, SGAN-E, SGAN-F) are explored, with SGAN-F delivering state-of-the-art quantitative restoration and SGAN-E excelling in perceptual quality; both synthetic and lab experiments validate the approach up to $10^6$ times the sensor saturation threshold. The method offers practical impact for protecting cameras in autonomous systems, HDR imaging, and laser safety devices, while maintaining high image fidelity in laser-free conditions and suggesting avenues for broadband and fully blind restoration in future work.

Abstract

Machine vision is susceptible to laser dazzle, where intense laser light can blind and distort its perception of the environment through oversaturation or permanent damage to sensor pixels. Here we employ a wavefront-coded phase mask to diffuse the energy of laser light and introduce a sandwich generative adversarial network (SGAN) to restore images from complex image degradations, such as varying laser-induced image saturation, mask-induced image blurring, unknown lighting conditions, and various noise corruptions. The SGAN architecture combines discriminative and generative methods by wrapping two GANs around a learnable image deconvolution module. In addition, we make use of Fourier feature representations to reduce the spectral bias of neural networks and improve its learning of high-frequency image details. End-to-end training includes the realistic physics-based synthesis of a large set of training data from publicly available images. We trained the SGAN to suppress the peak laser irradiance as high as $10^6$ times the sensor saturation threshold - the point at which camera sensors may experience damage without the mask. The trained model was evaluated on both a synthetic data set and data collected from the laboratory. The proposed image restoration model quantitatively and qualitatively outperforms state-of-the-art methods for a wide range of scene contents, laser powers, incident laser angles, ambient illumination strengths, and noise characteristics.

Figures (7)

• Figure 1: Laser dazzle protection at a glance. (a) Schematic of the wavefront-coded camera to diffuse laser irradiation. Quasi-monochromatic background illumination $\lambda_b$, which is close to the laser wavelength $\lambda_b$ is assumed. A five half-ring phase mask wirth2020half is placed adjacent to the lens. (b) Simulation of a phase-coded image of a background scene and potential sensor-damaging laser. (c) Restored image using our SGAN-F model and (d) Ground truth image.
• Figure 2: Overview of the physics-based modeling of a monochromatic anti-dazzle imaging system. (a) Image formation of background scene $b$ and a laser $l$ through a phase mask and a circular lens. (b) The imaging pipeline transforms the total radiance $b+l$ into a digitized image $s$ on the sensor. The convolution of the radiance and the system PSF results in an irradiance map $I$, which determines the rate at which photons arrive at the sensor. The sensor converts photons $\omega$ to electrons $e$ given a quantum efficiency $Q_{e}$, at which stage dark current $n_c$ and read noise $n_r$ are also generated. The total number of electrons that exceeds the full well capacity may experience saturation. The electrons are then scaled by a sensor gain $\mathcal{G}$ and quantized to an array of digital counts, the dimension of which is limited by the finite sensor size ($W_s, H_s$). The simulated irradiance distribution of (c) the uncoded PSF $h_0$ and (d) the phase-coded PSF $h$.
• Figure 3: The architecture of the neural sandwich GAN (SGAN) model, which consists of a set of generators $G = \{G_1, G_\mathrm{v}, G_2\}$ and a set of discriminators $D = \{D_1, D_2\}$. The U-shape generator $G_1$ removes noise, inpaints the laser contribution, and outpaints the cutoff image boundary from the concatenation of the zero-padded input image $s$ and its coordinates $(x,y)$, producing an estimated irradiance map $\hat{I}_b$. The deconvolution module $G_v$ extracts a set of features from the pre-restored image using the feature extractor $FE$. The feature images are individually deconvolved by the Deconv engine, where the noise power spectrum $\gamma$ is learnable. The U-shape generator $G_2$ combines and refines the deconvolved features into the restored images $\{\hat{b}^L| L = 0,1\}$ on a coarse scale ($L = 0$) and a fine scale ($L = 1$) respectively. The model is adversarially trained with a conditional discriminator $D_1$ and a multiscale discriminator $D_2 = \{D_2^L\}$ end-to-end. Three variances of the SGAN are investigated. They include the basic SGAN-B model, the enhanced SGAN-E model, and the frequency SGAN-F model. The SGAN-B/E models are built with basic residual, encode, and decoder blocks, while the SGAN-F model makes use of FFT representations in the encoders and decoders.
• Figure 4: Laboratory prototype of the anti-dazzle imaging system. A coherent laser source with wavelength $\lambda=633$ nm is redirected by a mirror $M_1$ and is expanded by a spatial filter $SF$. The laser light and the incoherently illuminated background scene simultaneously pass through a beam splitter $BS_1$, forming a joint light cone. The light cone is collimated by the first lens $L_1$, which is a focal length away from the scene with $f_1 = 40$ cm. A laser line filter is attached to the light emitting diode (LED) to produce quasi-monochromatic illumination with a central wavelength $\lambda$. The pupil is located at the second lens $L_2$ with $f_2 = 10$ cm. The lens $L_3$ has a focal length $f_3 = 10$ cm and is located at $20$ cm from $L_2$. It images the pupil to the SLM, which then retro-reflects the predetermined five half-ring phase pattern to $L_3$ and produces an engineered PSF at the pupil. The coded image is formed at the focal plane of the $L_2$ where unwanted reflection is blocked by a razor blade $RZ$. The intermediate image is magnified and reimaged on to a CCD sensor by a lens $L_4$ with $f_4=20$ cm and a mirror $M_2$. A circular field stop $FS$ between $L_2$ and $BS_2$ limits the field of view. The ground truth is recorded by turning off the laser source and the SLM.
• Figure 5: Evaluation of laser-dazzle protection in simulation. Our SGAN-B/E/F models are compared with alternative methods for image restoration of a laser-free case ($\alpha_l = 0$) in rows 1 and 2, and a damaging laser-dazzle case ($\alpha_l = \text{1e6}$) in rows 3 and 4. In each case, images restored by low- and high-performing models are respectively shown in the top and the bottom strips. MPRNet zamir2021multi, Pix2Pix isola2017image, and ST-CGAN wang2018stacked yield significantly distorted results in both cases. Uformer wang2022uformer, DeepWiener dong2020deep, WienerNet yanny2022deep, and our SGAN-B deliver reasonable recoveries, but perform poorly in the presence of laser dazzle. DeblurGAN kupyn2018deblurgan, Stripformer tsai2022stripformer, and MAXIM-2S tu2022maxim show improvements against laser dazzle in terms of coarse image features; however, fine image details (see the zoom-in image patches outlined by yellow boxes) remain unrecognizable regardless of the laser strengths. Without a laser, high-frequency features become recognizable in the images restored by MIMO-UNet cho2021rethinking, DeepRFT xint2023freqsel, and our SGAN-E. Among all, our SGAN-F produces the consistently highest fidelity image in both the laser-free and laser-dazzle cases.