Simulating Dual-Pixel Images From Ray Tracing For Depth Estimation

TL;DR

This work tackles the lack of real dual-pixel depth data by introducing Sdirt, a realism-oriented DP image simulation framework that combines a ray-traced, aberration-aware DP PSF simulator with a per-pixel PSF predictor. By rendering DP images from RGBD sources through per-pixel PSFs, Sdirt enables training of a depth-from-dual-pixel model that generalizes better to real DP data. The authors also collect the DP119 real-world test set to benchmark domain gap bridiging, and demonstrate that a DfDP model trained with Sdirt-based data achieves superior depth estimation across planar, box, and casual scenes. Overall, Sdirt provides a practical path to produce realistic DP data for smartphone, automotive, and microscope imaging scenarios where hardware constraints limit DP data collection. The approach emphasizes aberration and phase-splitting cues, enhancing the realism and transferability of learned depth estimation models.

Abstract

Many studies utilize dual-pixel (DP) sensor phase characteristics for various applications, such as depth estimation and deblurring. However, since the DP image features are entirely determined by the camera hardware, DP-depth paired datasets are very scarce, especially when performing depth estimation on customized cameras. To overcome this, studies simulate DP images using ideal optical system models. However, these simulations often violate real optical propagation laws, leading to poor generalization to real DP data. To address this, we investigate the domain gap between simulated and real DP data, and propose solutions using the Simulating DP images from ray tracing (Sdirt) scheme. The Sdirt generates realistic DP images via ray tracing and integrates them into the depth estimation training pipeline. Experimental results show that models trained with Sdirt-simulated images generalize better to real DP data. The code and collected datasets will be available at github.com/LinYark/Sdirt

Figures (15)

• Figure 1: (a) Imaging process of a DP camera. The slight shifts between the left and right DP images caused by phase differences are illustrated with white dashed lines. (b) Example comparison between real and CoC-simulated DP PSFs, showing a significant difference between them.
• Figure 2: Simulating Dual-Pixel Images from Ray Tracing pipeline. (a) Ray-traced DP PSF simulator. Calculates spatially varying DP PSFs for lens and DP sensor through ray tracing. (b) DP PSF prediction network. Trains an MLP network to predict DP PSFs, using the ray-traced DP PSFs as GT. (c) Pixel-wise DP image rendering module. The network predicts the DP PSFs for all points in the depth map (red pass). Then, each DP PSF is convolved with the AiF RGB image to render the simulated DP image (blue pass).
• Figure 3: (a) DP pixel structure layout. The left part is a right-side view, and the right part is a front perspective view. (b) On hitting the microlens, the $k$-th ray lands on left/right sub-pixel in red/blue interval, else missed. (c) Without hitting the microlens, the $k$-th ray lands on left/right sub-pixel in red/blue interval, else missed.
• Figure 4: (a) Derivation hints for \ref{['eq:f1_0']}. (b) The valid imaging region is a frustum, and we normalize the $(x,y)$ coordinates to $[-1,1]$. (c) The DfDP model takes DP image as input to predict the depth map. (d) During cost volume generation in cheng2020hierarchical, we stack original disparity (green arrows) and add reverse disparity (blue arrows), with $d_{max}$ as max displacement.
• Figure 5: Qualitative results of simulated DP PSFs. Evaluate the real and simulated (ours, CoC, L2R abuolaim2021learning, Modeling punnappurath2020modeling, and DDDNet pan2021dual) F/1.8 DP PSFs at two depths (0.5 m and 1.5 m) and three different positions. As the object point $p$ moves further from the optical axis, the real $PSF_L$ and $PSF_R$ become more phase asymmetric, and aberrations increase. Existing simulators neglect aberrations and DP phase splitting, causing a large gap between simulated and real DP PSFs. Only our ray-traced simulator predicts realistic results at all depths and positions.