Curriculum Learning for ab initio Deep Learned Refractive Optics

TL;DR

A DeepLens design method based on curriculum learning is presented, able to learn optical designs of compound lenses from randomly initialized surfaces without human intervention, demonstrating fully automated design of both classical imaging lenses and extended depth-of-field computational lenses.

Abstract

Deep optical optimization has recently emerged as a new paradigm for designing computational imaging systems using only the output image as the objective. However, it has been limited to either simple optical systems consisting of a single element such as a diffractive optical element (DOE) or metalens, or the fine-tuning of compound lenses from good initial designs. Here we present a DeepLens design method based on curriculum learning, which is able to learn optical designs of compound lenses ab initio from randomly initialized surfaces without human intervention, therefore overcoming the need for a good initial design. We demonstrate the effectiveness of our approach by fully automatically designing both classical imaging lenses and a large field-of-view extended depth-of-field computational lens in a cellphone-style form factor, with highly aspheric surfaces and a short back focal length.

Figures (3)

• Figure 1: Curriculum learning for automated lens design.a We utilize a differentiable ray-tracing approach to simulate the sensor captured image of an object image. This sensor capture can then be input into a downstream deep network for image reconstruction. During the forward image simulation (black arrows), we track the gradient of each optical parameter. We can subsequently back-propagate (blue arrows) the errors from either the simulated image for classical optical design, or from the network output for end-to-end optical design. The end-to-end optical design jointly optimizes the optical lens and the image reconstruction network. Classical lens design methods often face issues such as local minima and degenerate optical structures, including self-intersections, requiring appropriate starting points and consistent human intervention. We introduce a curriculum learning strategy that encompasses: a curriculum path (gray dashed arrow in a), optical regularization (b), and a re-weighting mask (c). b The optical regularization term presents lens from degenerate structures during the optimization. c The re-weighting mask dynamically directs attention towards problematic regions of simulated images during each epoch, compelling the optimization process to escape local minima. This curriculum learning strategy aims to automate the design of complex optical lenses from scratch, for both classical and computational lenses. d An example of this automated classical lens design using the curriculum learning strategy. The lens design process initiates from a flat structure, gradually elevating the design complexity until it meets the final design specifications. Detailed evaluations can be found in Table 1 and Supplementary Note 4.1. The cat image was photographed by Xinge Yang (CC BY 2.0).
• Figure 2: Evaluation of deep learned large-aperture EDoF lens.a The first surface of a classical aspherical lens is replaced by a hybrid odd-polynomial-aspheric surface to form an EDoF lens. The optical parameters of the EDoF lens are jointly optimized with the image reconstruction network in an end-to-end training manner. b For left to right: PSFs of the classical lens, EDoF lens with hybrid surface, and EDoF lens with an extra cubic plate at different depths and view angles. The PSFs of our deep learned EDoF lens are more depth-invariant compared to that of the classical lens. PSFs of more wavelengths are provided in Supplementary Note 5.3. c The height profile of the odd-polynomial term of the hybrid surface, which brings the EDoF ability to the lens system. d Image quality evaluation of simulated raw images and network reconstruction with PSNR and SSIM matrices. e MTF curves of the classical and EDoF lens at different depths, without image reconstruction. f Zoomed patches of network reconstructions at different depths. More evaluation results are provided in Supplementary Note 5.4 and 5.5.
• Figure 3: Distortion relaxation and adjoint rendering in differentiable ray tracing. Distortion relaxation: in large FoV optical length design, geometric distortion is usually relaxed for better control over other optical aberrations. The basic image-based loss function can not address this issue as it calculates per-pixel errors. To allow for geometric distortion in the final designed lens, we calculate the inverse distortion mapping relation and use it to pre-warp the object image for ray-tracing-based rendering. For example, if a lens has barrel distortion, we apply a pincushion distortion to the object image. Then during the image simulation, two distortions cancel out and the simulated sensor image is distortion-free compared to the ground truth. Adjoint rendering: high-resolution differentiable ray tracing consumes a large amount of memory. To address this issue, we propose an adjoint rendering approach that recalculates the ray-tracing simulation during backpropagation. This approach separates the gradient calculation of the network part (orange arrow) and the ray-tracing part (blue arrow), without compromising the calculation. To further improve the efficiency of our approach, we also incorporate a patch backpropagation method that recursively backpropagates gradients for image patches. The cat image was photographed by Xinge Yang (CC BY 2.0).