Fovea Stacking: Imaging with Dynamic Localized Aberration Correction

TL;DR

This paper introduces Fovea Stacking, a new type of imaging system that utilizes an emerging dynamic optical component called the deformable phase plate (DPP) for localized aberration correction anywhere on the image sensor and introduces a neural network-based control model for improved agreement between simulation and hardware performance.

Abstract

The desire for cameras with smaller form factors has recently lead to a push for exploring computational imaging systems with reduced optical complexity such as a smaller number of lens elements. Unfortunately such simplified optical systems usually suffer from severe aberrations, especially in off-axis regions, which can be difficult to correct purely in software. In this paper we introduce Fovea Stacking , a new type of imaging system that utilizes emerging dynamic optical components called deformable phase plates (DPPs) for localized aberration correction anywhere on the image sensor. By optimizing DPP deformations through a differentiable optical model, off-axis aberrations are corrected locally, producing a foveated image with enhanced sharpness at the fixation point - analogous to the eye's fovea. Stacking multiple such foveated images, each with a different fixation point, yields a composite image free from aberrations. To efficiently cover the entire field of view, we propose joint optimization of DPP deformations under imaging budget constraints. Due to the DPP device's non-linear behavior, we introduce a neural network-based control model for improved alignment between simulation-hardware performance. We further demonstrated that for extended depth-of-field imaging, fovea stacking outperforms traditional focus stacking in image quality. By integrating object detection or eye-tracking, the system can dynamically adjust the lens to track the object of interest-enabling real-time foveated video suitable for downstream applications such as surveillance or foveated virtual reality displays

Figures (18)

• Figure 1: (a) physical layout of DPP device, composed of a deformable membrane, a optical liquid cavity, and a rigid glass substrate containing hexagonal electrodes. Figure adapted from rajaeipour2021seventh. (b) localized aberration of different oblique angles can be corrected by dynamically changing DPP patterns. The green-channel wavefront aberration on exit-pupil plane and the corresponding colored spot diagrams are shown.
• Figure 2: Side view of the calibrated simulation system. The object plane is set 652 mm away. The imaging setup includes a DPP, a 50 mm achromatic doublet lens (Thorlabs AC-254-050A), and a Bayer pattern RGB sensor (FLIR GS3-U3-41C6C-C).
• Figure 3: Optimized PSF and MTF for various oblique angles $\phi$ relative to the optical axis: (a) initial system without DPP, (b) optimization of the defocus Zernike parameter only, simulating improvements that can be made by focus adjustments (either by shifting the lens or with a liquid tunable lens) and (c) full Zernike optimization up to 4th order. MTF values are averaged over sagittal and tangential components.
• Figure 4: Radial and angular coverage of the fovea area. (a) For a single optimized DPP pattern, the fovea area—defined as the area where MTF50 exceeds a threshold like 30 lp/mm—can be characterized by its extent along the normalized radial axis $\rho$ and angular axis $\theta$. The gray square indicates the sensor region. (b) As the optimized angle approaches the sensor's edge, the fovea area shrinks, necessitating more images in the peripheral areas to maintain sharpness over the gray area representing the sensor.
• Figure 5: Joint optimization of 5 DPP patterns using grid stacking. (b) After optimization, each DPP deformation is specialized to optimize quality over different local regions while jointly covering the full FoV. (a) As an example the $1^{st}$ DPP pattern produces the best focus in the top-left and bottom-right regions. (c) The minimal value of RMS spot size $r_{min}$ across the stack serves as the optimization loss, while the mask index $n^*$ indicates which DPP pattern is best for any given region, which can be used to generate (d) a composite PSF map for the image after stacking.