Accurate Eye Tracking from Dense 3D Surface Reconstructions using Single-Shot Deflectometry

TL;DR

A method for accurate and fast evaluation of the gaze direction that exploits teachings from single-shot phase-measuring-deflectometry, leading to an increase in acquired surface data points by factors > 3000X compared to conventional methods.

Abstract

Eye-tracking plays a crucial role in the development of virtual reality devices, neuroscience research, and psychology. Despite its significance in numerous applications, achieving an accurate, robust, and fast eye-tracking solution remains a considerable challenge for current state-of-the-art methods. While existing reflection-based techniques (e.g., "glint tracking") are considered to be very accurate, their performance is limited by their reliance on sparse 3D surface data acquired solely from the cornea surface. In this paper, we rethink the way how specular reflections can be used for eye tracking: We propose a novel method for accurate and fast evaluation of the gaze direction that exploits teachings from single-shot phase-measuring-deflectometry(PMD). In contrast to state-of-the-art reflection-based methods, our method acquires dense 3D surface information of both cornea and sclera within only one single camera frame (single-shot). For a typical measurement, we acquire $>3000 \times$ more surface reflection points ("glints") than conventional methods. We show the feasibility of our approach with experimentally evaluated gaze errors on a realistic model eye below only $0.12^\circ$. Moreover, we demonstrate quantitative measurements on real human eyes in vivo, reaching accuracy values between only $0.46^\circ$ and $0.97^\circ$.

Figures (7)

• Figure 1: Deflectometry-based eye-tracking (a) Schematic of our single-shot deflectometry-based gaze estimation approach: Two cameras observe the specular reflection of a display over the eye surface. Eye surface shape and normal information are calculated from the deformation of the pattern in the camera images. (b) Simulation: For a perfectly spherical cornea and sclera and no noise, cornea and sclera center can be obtained by tracing the measured surface normals back towards the eye center. The cornea and sclera center points $O_c, O_s$ can be used to obtain the optical axis and gaze direction. (c) Real experiment: If cornea and sclera are not spherical, but rotational symmetric, all back traced normals still intersect along the optical axis.
• Figure 2: Single-shot and multi-shot deflectometry for eye surface measurement. (a) Schematic of standard deflectometry: a display with a known pattern illuminates the specular object. The camera observes the deformation of the pattern after reflection. (b) Single-shot deflectometry raw data: Realistic eye model with the reflection of the cross-sinusoid pattern. (c) and (d) Respective retrieved vertical phase map and unwrapped display-camera correspondence map. (e) Multi-shot deflectometry raw data for comparison: Realistic eye model with reflection of a sinusoidal pattern with horizontal stripes. (f) The four-phase shifting method can be used to retrieve the phase map (g) and eventually the display-camera correspondence map (h) after unwrapping.
• Figure 3: Novel algorithm for surface reconstruction and gaze estimation. (a,b) Initial surface reconstruction: (a) We use a few points in the initial stereo-deflectometry overlap region to estimate the initial radii $R_c', R_s'$ and centers $O_c', O_s'$ of cornea and sclera to construct an initial 2-sphere model. The normals $\overrightarrow{n}$ calculated via deflectometry do not match with the normals $\overrightarrow{n_s}$ of the initial sphere models. (b) We optimize the centers and radii of the initial two-sphere model by expanding the evaluation region to the whole measurement area of both cameras (the red part is the overlap region, orange part is covered by one camera only). New estimates $R_c", R_s", O_c", O_s"$ for radii and centers are obtained by minimizing the angular difference between all pairs of $\overrightarrow{n}$ and $\overrightarrow{n_s}$ (see Eq. \ref{['eq:important']}). (c,d) Refinement: (c) We use the obtained 2-sphere model as initial guess and calculate a new surface representation via iterative deflectometry normal integrationslogsnat2009nonhuang2018reviewhuang2017zonal. After this step, the resulting surface parts are not necessarily spherical anymore (in accordance with real human eyes). (d) When back traced, the newly obtained normals do not meet at two distinct center points anymore. However, assuming the human eye is rotationally symmetric, the normals still meet along its optical axis, which is updated accordingly and used to calculate the gaze.
• Figure 4: Novel method for calibration of deflectometry setups (a) and (b) Shortcomings of the "classical" methodli2021lowwang2022easy: The display is observed over an auxiliary mirror, whose position in space is determined by attached markers. For precise pose estimation, the markers need to cover a large part of the available FoV, which occludes the display and/or restricts the effective measurement field. (c) and (d) Novel calibration method: A specular phone screen replaces the mirror. In the first step, a dense marker pattern is displayed on the phone screen, which is used to determine the screen's position in space (c). Eventually, the screen is switched off and functions as a mirror that reflects the display and hence allows for a full-field calibration of the whole display without occlusions (d). The upper right corner of (c) and (d) are camera images of the captured "marker" and "display reflection".
• Figure 5: Quantitative evaluation of surface reconstruction quality. (a) The system accuracy is evaluated by measuring a spherical bearing ball with the known radius (close to the radius of a real eye). (b) Reconstructed ball surface with measured normal vectors. The calculated radius is 12.02mm, with the ground truth radius being 12mm. The standard deviation of the distance of the calculated sphere center to all normals is $62\mu m$.