Towards Single-Lens Controllable Depth-of-Field Imaging via Depth-Aware Point Spread Functions

TL;DR

This paper tackles achieving controllable depth-of-field imaging with a single minimalist optical system by learning to recover an aberration-free AiF image and a depth map, then synthesizing lens-specific DoF via depth-aware PSFs. The authors introduce the DAMOS dataset and the DA$^{2}$T training scheme to capture depth-dependent degradation, along with two depth-aware blocks (RDICAB and D$^{2}$CB) to improve aberration correction. An Omni-Lens-Field model compactly represents a 4D PSF library for multiple lenses, enabling efficient, depth-aware DoF rendering through patch-wise convolution. Experiments on simulated and real data show improved restoration quality and impressive controllable DoF results, establishing a strong baseline for MOS-based computational imaging.

Abstract

Controllable Depth-of-Field (DoF) imaging commonly produces amazing visual effects based on heavy and expensive high-end lenses. However, confronted with the increasing demand for mobile scenarios, it is desirable to achieve a lightweight solution with Minimalist Optical Systems (MOS). This work centers around two major limitations of MOS, i.e., the severe optical aberrations and uncontrollable DoF, for achieving single-lens controllable DoF imaging via computational methods. A Depth-aware Controllable DoF Imaging (DCDI) framework is proposed equipped with All-in-Focus (AiF) aberration correction and monocular depth estimation, where the recovered image and corresponding depth map are utilized to produce imaging results under diverse DoFs of any high-end lens via patch-wise convolution. To address the depth-varying optical degradation, we introduce a Depth-aware Degradation-adaptive Training (DA2T) scheme. At the dataset level, a Depth-aware Aberration MOS (DAMOS) dataset is established based on the simulation of Point Spread Functions (PSFs) under different object distances. Additionally, we design two plug-and-play depth-aware mechanisms to embed depth information into the aberration image recovery for better tackling depth-aware degradation. Furthermore, we propose a storage-efficient Omni-Lens-Field model to represent the 4D PSF library of various lenses. With the predicted depth map, recovered image, and depth-aware PSF map inferred by Omni-Lens-Field, single-lens controllable DoF imaging is achieved. Comprehensive experimental results demonstrate that the proposed framework enhances the recovery performance, and attains impressive single-lens controllable DoF imaging results, providing a seminal baseline for this field. The source code and the established dataset will be publicly available at https://github.com/XiaolongQian/DCDI.

Figures (19)

• Figure 1: Post-capture controllable Depth-of-Field (DoF) Imaging. (a) Aberration image captured by the MOS-S1; (b) Predicted depth map from UniDepth piccinelli2024unidepth; (c) Predicted All-in-Focus (AiF) image recovered by our Depth-aware Aberration Correction Network (DACN); (d) Controllable DoF imaging. Our method can produce diverse DoF imaging effects of different lenses. We show our real-world results, keeping the nearest doll in the scene clear while generating bokeh effects of two different lenses (MOS-S1 and MOS-S2). Detail regions are highlighted at different depths and the corresponding PSFs, which are amplified with colored boxes. Please zoom in for the best view.
• Figure 2: Overview of the proposed Depth-aware Controllable DoF Imaging (DCDI) framework. (a) We simulate the Depth-aware Aberration MOS (DAMOS) dataset. The Depth-aware 4D PSFLib is constructed by performing multiple ray tracing simulations, varying the position of the object plane from near to far. (b) The MDE model predicts scene depth map. (c) The latent AiF image is recovered by jointly combining synthetic data, predicted depth map, and depth-aware network architecture. (d) Controllable DoF imaging of multiple lenses is achieved through predicted depth maps, restored AiF images, and depth-aware PSF maps predicted by Omni-Lens-Field.
• Figure 3: Overview of the proposed Depth-aware Aberration Correction Network (DACN). DACN is established on a classical super-resolution paradigm, which includes shallow feature extraction, deep feature extraction, and reconstruction. RDICAB and D$^{2}$CB are two proposed plug-and-play depth-aware mechanisms that can effectively learn depth-aware degradation with the guidance of depth features. Here, RACM denotes the Replaceable Aberration Correction Module, which implies that any suitable restoration module can be utilized.
• Figure 4: Illustration of the proposed Omni-Lens-Field model training procedure. The network is trained to represent the 4D PSFLib of various lenses. The network takes the normalized coordinates of the image patch on the object plane $(c,h,w)$ and the lens ID $N$ as input and outputs a 3D tensor as the predicted value.
• Figure 5: we manufacture (a) MOS-S1 and mount it on a (b) Sony ${\alpha}$6600 camera to capture the real-world dataset.