Learned split-spectrum metalens for obstruction-free broadband imaging in the visible

TL;DR

The paper addresses obstruction in imaging by introducing a learned split-spectrum metalens for broadband obstruction-free imaging, supported by a depth–wavelength symmetry model with $z = lambda_d f / (lambda - lambda_d)$ and a wavelength shift relation $Delta lambda = lambda_d f (1/z - 1/(z-Delta z))$. It achieves obstruction-free broadband imaging by partitioning each RGB channel into pass/stop bands and training a differentiable metalens–vision pipeline to focus far-depth light while suppressing near-depth obstructions, yielding sharp far PSFs and defocused near PSFs. The approach delivers PSNR gains up to $32.29\%$ over a conventional hyperbolic metalens and provides substantial improvements in downstream vision metrics (e.g., mAP, IoU, mIoU) on VisDrone, Kvasir-SEG, and Cityscapes, while preserving high fidelity for unobstructed scenes. These results demonstrate robust, compact obstruction-free sensing suitable for mobile robots, drones, and endoscopic probes, with analytical, fabrication, and end-to-end imaging contributions advancing practical diffractive optics.

Abstract

Obstructions such as raindrops, fences, or dust degrade captured images, especially when mechanical cleaning is infeasible. Conventional solutions to obstructions rely on a bulky compound optics array or computational inpainting, which compromise compactness or fidelity. Metalenses composed of subwavelength meta-atoms promise compact imaging, but simultaneous achievement of broadband and obstruction-free imaging remains a challenge, since a metalens that images distant scenes across a broadband spectrum cannot properly defocus near-depth occlusions. Here, we introduce a learned split-spectrum metalens that enables broadband obstruction-free imaging. Our approach divides the spectrum of each RGB channel into pass and stop bands with multi-band spectral filtering and learns the metalens to focus light from far objects through pass bands, while filtering focused near-depth light through stop bands. This optical signal is further enhanced using a neural network. Our learned split-spectrum metalens achieves broadband and obstruction-free imaging with relative PSNR gains of 32.29% and improves object detection and semantic segmentation accuracies with absolute gains of +13.54% mAP, +48.45% IoU, and +20.35% mIoU over a conventional hyperbolic design. This promises robust obstruction-free sensing and vision for space-constrained systems, such as mobile robots, drones, and endoscopes.

Figures (6)

• Figure 1: Learned split-spectrum metalens for obstruction-free broadband imaging. Our learned metalens enables obstruction-free broadband imaging by filtering out the focused light from near depth with a multi-band spectral filter that selectively transmits far-focused light.
• Figure 2: Depth-wavelength symmetry.a Schematic of the depth-wavelength relationship of diffractive lenses. (Left) An incident wavelength $\lambda$ longer than the design wavelength $\lambda_\text{d}$ ($\lambda>\lambda_\text{d}$) of the lens causes a phase mismatch, resulting in a focal front shift. (Right) This focal front shift can be compensated for by the spherical phase of an incident wave originating from a near depth, leading to an opposing focal back shift. b Focal point intensity map for point light sources of wavelength, $\lambda$, and depth, $z$, incident on a hyperbolic ($\lambda_\text{d}=450\,$nm) metalens, computed with Rayleigh-Sommerfeld diffraction, and overlaid with our depth-wavelength symmetry model.
• Figure 3: Learning split-spectrum metalens for obstruction-free broadband imaging.a Overview of the metalens optimization pipeline. The metalens design map $\theta(x,y)$ is learned for the obstruction-free broadband imaging. b Transmission of the multi-band spectral filter, separating each color channel into the pass band $\Lambda_\text{pass}^{c}$ and the stop band $\Lambda_\text{stop}^{c}$, providing extended design space for obstruction-free imaging. c Our imaging system consists of a metalens, a color sensor, and a multi-band spectral filter. d$x-\lambda$ scan result of the learned metalens for far PSFs and near PSFs in simulation. e Training loss and visual convergence during learning. The sequence of the simulated sensor images (step 50, 200, 3200) illustrates the learning of the metalens. Initial results exhibit significant blur and interference from obstructions, while the final optimized metalens effectively blurs the near-depth obstructions and maintains a sharp focus on the far-depth scene.
• Figure 4: Characterization of the learned split-spectrum metalens.a Scanning electron microscope (SEM) images of the learned split-spectrum metalens with the top-down view (left) and tilted view (right). b Photograph of the three fabricated metalenses. From left to right: the learned split-spectrum metalens (Ours), the learned broadband metalens without the split-spectrum strategy (Broadband), and the conventional hyperbolic-phase metalens (Hyperbolic). c PSF $x-\lambda$ scan results normalized to the total intensity. Pass bands are highlighted. Far-depth focused lights of the obstruction-free metalens (Ours) can be transmitted and captured on the sensor, while near-depth focused lights are filtered out, enabling obstruction-free imaging. d 2D PSFs of metalenses for different depths, normalized to their peak intensities. The stop band PSFs of Ours are marked with black boxes. c, d Our far-depth PSFs remain sharp in the pass bands, enabling clear imaging of the originally occluded scene, while the near-depth PSFs are blurry. The hyperbolic metalens suffers from severe chromatic aberrations. The broadband metalens fails to achieve the near-depth blur required for obstruction-free imaging.
• Figure 5: Experimental evaluation of obstruction-free broadband imaging.a Near-depth obstruction pattern used for the experiment. b Raw sensor captured images under the obstruction for the three metalens designs. Insets provide a magnified view of the regions highlighted by yellow and red boxes. c Ground-truth (GT) unobstructed reference image captured with a compound lens with $f=8\,$mm (two times longer than those of metalenses) for better object plane resolution. d Reconstructed images using the neural network trained for each metalens. e Reconstructed image comparisons of representative printed 2D image captures under fence and dirt obstructions. f PSNRs measured on the printed image dataset. b, d, e, f Our design suppresses obstructions better than other metalens baselines, achieving superior performance across all conditions.