Passive Snapshot Coded Aperture Dual-Pixel RGB-D Imaging

TL;DR

CADS introduces a passive, single-shot RGB-D imaging approach by pairing a dual-pixel sensor with a learnable coded aperture. The method jointly optimizes the aperture mask and a neural reconstruction network (CADNet) in an end-to-end framework, using a differentiable DP-PSF model and MPI-based rendering to produce accurate depth maps and all-in-focus images across a range of apertures. Across simulations and real-world prototypes (DSLR, endoscopy, dermoscopy), CADS achieves meaningful gains in AIF PSNR (over 1.5 dB) and depth accuracy (5–6%), while maintaining a compact, snapshot-capable footprint. The work demonstrates CADS' practical impact for fields requiring small form factors and fast 3D sensing, including medical imaging and consumer photography, and points to future improvements via phase-mask alternatives and broader PSF modeling.

Abstract

Passive, compact, single-shot 3D sensing is useful in many application areas such as microscopy, medical imaging, surgical navigation, and autonomous driving where form factor, time, and power constraints can exist. Obtaining RGB-D scene information over a short imaging distance, in an ultra-compact form factor, and in a passive, snapshot manner is challenging. Dual-pixel (DP) sensors are a potential solution to achieve the same. DP sensors collect light rays from two different halves of the lens in two interleaved pixel arrays, thus capturing two slightly different views of the scene, like a stereo camera system. However, imaging with a DP sensor implies that the defocus blur size is directly proportional to the disparity seen between the views. This creates a trade-off between disparity estimation vs. deblurring accuracy. To improve this trade-off effect, we propose CADS (Coded Aperture Dual-Pixel Sensing), in which we use a coded aperture in the imaging lens along with a DP sensor. In our approach, we jointly learn an optimal coded pattern and the reconstruction algorithm in an end-to-end optimization setting. Our resulting CADS imaging system demonstrates improvement of >1.5dB PSNR in all-in-focus (AIF) estimates and 5-6% in depth estimation quality over naive DP sensing for a wide range of aperture settings. Furthermore, we build the proposed CADS prototypes for DSLR photography settings and in an endoscope and a dermoscope form factor. Our novel coded dual-pixel sensing approach demonstrates accurate RGB-D reconstruction results in simulations and real-world experiments in a passive, snapshot, and compact manner.

Figures (17)

• Figure 1: (A). We propose CADS - an imaging approach that leverages DP sensors and coded aperture masks for passive snapshot 3D imaging. (B) Coded DP sensing improves on naive dual-pixel sensing for simultaneous depth estimation and deblurring. Our CADS imaging prototype can recover accurate depth maps and all-in-focus images in (C) DSLR photography settings, as well as for (D) 3D endoscopy and dermoscopy. Read more on our website https://shadowfax11.github.io/cads/.
• Figure 2: Pipeline for Coded Aperture Dual-Pixel Sensing (CADS) and corresponding 3D scene estimation. We perform end-to-end (E2E) optimization on simulated data, to learn an optimal amplitude mask and neural network weights for predicting deblurred all-in-focus (AIF) images and depth maps from coded dual-pixel captures. Our end-to-end learned system provides the best trade-off between depth estimation and AIF quality.
• Figure 3: CADS PSF formation model. (A) With a DP sensor, scene points that are out-of-focus show disparity when comparing between left and right views. (B) This disparity is bi-directional, depending on scene depth relative to the in-focus plane. (C) When a coded amplitude mask is added to the imaging lens, the DP PSFs take the shape of the mask pattern. (D) Coded DP PSFs are a Hadamard product of the naive (no code) DP PSFs and the mask pattern.
• Figure 4: Comparison with naive standard and DP - simulation. DP sensing methods provide high-fidelity depth and all-in-focus (AIF) prediction compared to standard pixel sensing. Among DP sensing methods, CADS offers the best AIF and depth estimation quality observed through sharper AIF and cleaner depth.
• Figure 5: Comparison with naive standard and DP - real data. From left to right we show AIF and depth predictions by naive standard-pixel, naive dual-pixel and CADS, respectively, along with ground truth. CADS gives the best depth and AIF predictions. GT for AIF is obtained by capturing the scene with a $f/22$ aperture. A coarse GT depth map is obtained using the Intel RealSense sensor. Zoom in for best viewing.