Successive optimization of optics and post-processing with differentiable coherent PSF operator and field information

TL;DR

The paper tackles the challenge of jointly optimizing optics and post-processing for compact cameras where wavefront aberrations and diffraction dominate. It introduces a differentiable optical simulation with a coherent PSF operator and a memory-efficient backpropagation scheme, along with a robust Newton initialization strategy for reliable intersections on highly aspherical surfaces. A field-aware end-to-end pipeline couples a differentiable optical model with a MIMO-UNet restoration network, incorporating optical constraints and a joint loss $L_{joint}=L_{net}+\{lambda}_{lens}L_{optic}$. Experimental results demonstrate accurate PSF calculations close to Zemax ground truth, substantial memory savings, and progressive improvements in image quality and EMTF across multiple lenses, highlighting the practical impact for advanced compact-lens design; code will be publicly released.

Abstract

Recently, the joint design of optical systems and downstream algorithms is showing significant potential. However, existing rays-described methods are limited to optimizing geometric degradation, making it difficult to fully represent the optical characteristics of complex, miniaturized lenses constrained by wavefront aberration or diffraction effects. In this work, we introduce a precise optical simulation model, and every operation in pipeline is differentiable. This model employs a novel initial value strategy to enhance the reliability of intersection calculation on high aspherics. Moreover, it utilizes a differential operator to reduce memory consumption during coherent point spread function calculations. To efficiently address various degradation, we design a joint optimization procedure that leverages field information. Guided by a general restoration network, the proposed method not only enhances the image quality, but also successively improves the optical performance across multiple lenses that are already in professional level. This joint optimization pipeline offers innovative insights into the practical design of sophisticated optical systems and post-processing algorithms. The source code will be made publicly available at https://github.com/Zrr-ZJU/Successive-optimization

Figures (11)

• Figure 1: Schematic diagram of this work. In the left figure, we focus on successively optimizing the optical and image performance of advanced complex lens using a joint approach. Compared to the existing joint design paradigm, our key features are highlighted in the right flowchart.
• Figure 2: Overview of our joint optimization pipeline. In each iteration, a random object field is sampled to construct the input for the downstream restoration network, consisting of the blurred image patch and the normalized pixel coordinates. The lens parameters $\phi_{lens}$ are optimized based on the PSF loss derived from reconstruction errors and on optical evaluations, which consider both optical performance and geometrical constraints through exact ray tracing. Concurrently, the reconstruction network parameters $\phi_{net}$ are trained to minimize the reconstruction loss while adapting to the spatially varying lens aberrations.
• Figure 3: (a) shows the differences between our proposed initial value estimation strategy (blue) and the simple estimation method (red), with initial points marked by stars. (b) provides a schematic diagram of the coherent PSF calculation process. Note that the spacing between the grid points and the ray trace points is exaggerated for visualization purposes.
• Figure 4: Comparison of the gradient computation flow. (a) demonstrates the normal calculation procedure in automatic differentiation functions. (b) shows that the proposed differential operator manually back propagates the analytical gradients without saving the large broadcast tensors.
• Figure 5: Overview of the lenses specifications and layouts in the experiments. The relative difficulty of each lens design is estimated based on the optical Lagrange invariant, $nyu$. All of these lenses are utilized for PSF validation in Sec. \ref{['sec:psf valid']}, with the top-left three lenses also being used for joint optimization in Sec. \ref{['sec:joint']}.