Enabling High-Quality In-the-Wild Imaging from Severely Aberrated Metalens Bursts

TL;DR

The paper tackles high-quality in-the-wild imaging using ultra-thin metalenses by pairing a compact, broadband, achromatic metalens with a lightweight, burst-based restoration pipeline. It introduces a modular, end-to-end system that decouples restoration and HDR fusion, featuring an Adaptive Pixel Correction Unit, a multi-scale burst alignment strategy, and cross-attention-based fusion, enabling real-time edge-device performance without requiring paired metalens training data. Key contributions include a physically informed image formation model, a robust two-branch restoration network, unsupervised in-the-wild adaptation, and comprehensive evaluations showing superiority over state-of-the-art methods in both controlled and real-world handheld scenarios. The results demonstrate a practical route to deploy metalens-based cameras in everyday imaging tasks, with potential impact on AR/VR, wearables, and IoT devices by enabling ultra-compact, high-quality imaging hardware.

Abstract

We tackle the challenge of robust, in-the-wild imaging using ultra-thin nanophotonic metalens cameras. Meta-lenses, composed of planar arrays of nanoscale scatterers, promise dramatic reductions in size and weight compared to conventional refractive optics. However, severe chromatic aberration, pronounced light scattering, narrow spectral bandwidth, and low light efficiency continue to limit their practical adoption. In this work, we present an end-to-end solution for in-the-wild imaging that pairs a metalens several times thinner than conventional optics with a bespoke multi-image restoration framework optimized for practical metalens cameras. Our method centers on a lightweight convolutional network paired with a memory-efficient burst fusion algorithm that adaptively corrects noise, saturation clipping, and lens-induced distortions across rapid sequences of extremely degraded metalens captures. Extensive experiments on diverse, real-world handheld captures demonstrate that our approach consistently outperforms existing burst-mode and single-image restoration techniques.These results point toward a practical route for deploying metalens-based cameras in everyday imaging applications.

Figures (13)

• Figure 1: Compact Metalens Camera with Real-Time Edge Performance.(a) We jointly build a metalens, orders of magnitude thinner than a conventional camera lens, and a real-time burst image restoration method. Our camera module runs inference on a Jetson Nano Orin edge device. (b) We demonstrate in-the-wild imaging, including HDR reconstruction, as an avenue for our burst restoration pipeline. (c) Our real-time image restoration on the handheld metalens camera outperforms prior state-of-the-art methods.
• Figure 2: Metalens Optimization and Deblurring Artifacts.(a) We optimize the metalens phase profile via a radial parameterization, differentiable wave propagation, and joint focal spot (optical) and pixel-space (L2) losses. (b) Naive deconvolution around saturated areas produces ringing and halo artifacts.
• Figure 3: Pipeline Overview. Captured metalens bursts first go through a reference frame selector to identify the reference frame index ($r$). Each burst frame $I^i$ is then aligned to the reference, producing $I^i_\text{aligned}$. These aligned frames enter a weighted burst fusion module to produce a single fused image $I_\text{fused}$. An Adaptive Pixel Correction Unit (APCU) adjusts pixel intensities using a weighting operation, yielding $I_\text{init}$, which is finally refined by the restoration module to produce a high-quality output $I_\text{out}$.
• Figure 4: Burst Alignment. At each pyramid level, we first sharpen features via deconvolution before performing SIFT-based feature matching. We then compute local patch homographies and fuse them across neighboring patches and scales to achieve precise frame registration.
• Figure 5: Selective Feature Alignment. Our feature alignment module extracts and integrates burst-frame features conditioned on the reference frame to correct residual defects in the initial estimate $I_\text{init}$.