Task-Driven Lens Design

TL;DR

This work proposes task-driven lens design, a new optimization philosophy for joint optics-network systems that freezes the pretrained vision model and optimize only the lens so that the image formation better fits the model's feature preferences, enabling lens design from scratch without human intervention.

Abstract

Classical lens design minimizes optical aberrations to produce sharp images, but is typically decoupled from downstream computer vision tasks. Existing end-to-end optical design learns optical encoding through joint optimization, but often suffers from an unstable training process. We propose task-driven lens design, a new optimization philosophy for joint optics-network systems. We freeze the pretrained vision model and optimize only the lens so that the image formation better fits the model's feature preferences. This network-frozen setting yields a low-dimensional and stable optimization process, enabling lens design from scratch without human intervention, thereby exploring a broader design space. Multiple computer vision experiments show that TaskLenses outperform classical ImagingLenses with the same or even fewer elements. Our analysis reveals that the learned optics exhibit long-tailed point spread functions, better preserving preferred structural cues when aberrations cannot be fully corrected. These results highlight task-driven design as a practical route for optical lenses that are compatible with modern vision models, and also inspire new optical design objectives beyond traditional aberration minimization.

Figures (3)

• Figure 1: The task-driven lens design pipeline begins by calculating the PSF via differentiable ray tracing. This PSF is then convolved with input images to simulate camera captures. A well-trained computer vision network (e.g., for image classification) evaluates the final output error and backpropagates the loss function to optimize lens parameters. Unlike classical lens design, which minimizes optical aberrations, task-driven lens design prioritizes capturing image features that are most effective for downstream computer vision networks. After optimization, the designed TaskLens better preserves image structure details even in the presence of optical aberrations, leading to improved performance on the target computer vision task. In contrast, an ImagingLens, designed to produce clear captures, is more susceptible to performance degradation from optical aberrations, potentially resulting in misclassification.
• Figure 2: The lens layout, PSFs and corresponding spot diagrams at different fields are shown for 2P (a), 3P (b), and 4P (c) lenses. Although RMS spot size of the TaskLens at some fields is larger than that of the ImagingLens, the majority of optical rays always converge within a small region. This spot diagram distribution results in a long-tailed PSF, which is characterized by a small, concentrated center and sparsely populated outer regions. In scenarios where the optical system cannot fully correct all aberrations, this long-tailed PSF with a sharp central peak is effective in preserving essential features of the object images, which benefits the performance of computer vision models.
• Figure 3: Validation of the optical simulation pipeline using a Canon EOS R6 and RF50mm f/1.8 lens. The experimental setup utilizes a test chart for MTF calculation and a monitor-displayed point source for PSF measurement. The measured MTF curves (top) track the simulated values closely, confirming accurate frequency response modeling. The captured PSFs (bottom) at three FoV exhibit strong morphological agreement with the simulation, validating the modeling of field-dependent aberrations.