Pupil-Adaptive 3D Holography Beyond Coherent Depth-of-Field

TL;DR

This work tackles the gap between coherent holographic depth-of-field and real-world incoherent defocus cues by incorporating continuous pupil-size variation into 3D holography. It introduces a unified pupil-adaptive framework that conditions hologram synthesis on current pupil size using an adjustable deformable convolution to dynamically vary receptive fields and depth-of-field. Training leverages photorealistic focal stacks and focal-stack supervision, enabling realistic pupil-dependent defocus across varying depths and pupil conditions, validated in both simulation and hardware with notable PSNR gains and qualitative improvements. The approach offers a practical path toward photo-realistic near-eye holography for AR/VR, while acknowledging limitations such as non-real-time performance and the need for eye-tracking in real deployments.

Abstract

Recent holographic display approaches propelled by deep learning have shown remarkable success in enabling high-fidelity holographic projections. However, these displays have still not been able to demonstrate realistic focus cues, and a major gap still remains between the defocus effects possible with a coherent light-based holographic display and those exhibited by incoherent light in the real world. Moreover, existing methods have not considered the effects of the observer's eye pupil size variations on the perceived quality of 3D projections, especially on the defocus blur due to varying depth-of-field of the eye. In this work, we propose a framework that bridges the gap between the coherent depth-of-field of holographic displays and what is seen in the real world due to incoherent light. To this end, we investigate the effect of varying shape and motion of the eye pupil on the quality of holographic projections, and devise a method that changes the depth-of-the-field of holographic projections dynamically in a pupil-adaptive manner. Specifically, we introduce a learning framework that adjusts the receptive fields on-the-go based on the current state of the observer's eye pupil to produce image effects that otherwise are not possible in current computer-generated holography approaches. We validate the proposed method both in simulations and on an experimental prototype holographic display, and demonstrate significant improvements in the depiction of depth-of-field effects, outperforming existing approaches both qualitatively and quantitatively by at least 5 dB in peak signal-to-noise ratio.

Figures (18)

• Figure 1: Evaluation of smooth phase and random phase holograms in simulation. We simulate holographic reconstructions of smooth and random phase holograms with their respective state-of-the-art methods. For the smooth phase holograms shi2022light, the light energy is concentrated at the center of the eyebox and fails to reproduce correct pupil-dependent depth-of-field effects. On the other hand, random phase holograms b-sgd achieves uniform energy distribution across the eyebox and correct defocus effects for different pupil sizes, but suffers from severe speckle noise, more notably for smaller pupil sizes. Please see \ref{['sec:motivation']} for a detailed discussion.
• Figure 3: Simulated reconstruction from the optimized hologram. (a) Reconstructed images at three focal planes for a 3mm-puil and 4-pupil at the center view. In-focus regions are highlighted in red and out-of-focus regions are in blue. (b) Illustration of the parallax effects. Insets are taken from reconstructions at middle focal plane at $9$ viewpoints with a 4mm-pupil.
• Figure 4: Eyebox pupil sampling.
• Figure 5: Pupil-aware light field holography with different pupil sampling intervals. We show the optimized holograms (green channel) in column (a) and corresponding energy distributions in (b). A full set of the reconstructed views (at middle depth plane) for a 3-mm pupil are shown in (c) and for a 4-mm pupil in (d). The insets in (e) highlights an enlarged region of the reconstructed center view for a 4-mm pupil. As the pupil interval $z$ increases, the phase pattern exhibits increased randomness and a broad eyebox energy distribution. However, this results in a gradual decline in the quality of reconstructed images.
• Figure 6: When a human eye is focusing at a distance (e.g., the middle plane), an object point appearing away from the focal plane will occur dilated (within an area) on the retina rather than as a clear point. The area on the retina is called the circle of confusion (CoC) and its diameter is proportional to the distance between the object point and the focal plane.