Compact Eye Tracking for VR/AR Displays via Deep Learned MicroLED Projection and Single-Pixel Sensing

TL;DR

This work tackles the challenge of high-speed, accurate eye tracking in compact VR/AR headsets by combining a lightweight microLED projection array with single-pixel sensing and two pattern bases: Hadamard and deep learned illumination patterns. A neural-network framework performs both gaze-region categorisation and precise regression of eye angles, trained with Hadamard-derived data and then finetuned on real measurements; the deep learned patterns achieve markedly better accuracy using only 72 patterns versus 480 Hadamard patterns. The system demonstrates angular accuracy better than $1^\ ext{degree}$ and a maximum measurement rate of $3.59\,\text{kHz}$ on a model eye, with live tracking showcasing rapid gaze estimation suitable for high-refresh VR/AR displays. The hardware is compact enough for headset integration and supports potential foveated rendering and responsive user interaction, marking a practical step toward fast, camera-free gaze tracking in head-worn displays.

Abstract

Fast and accurate eye tracking in a virtual reality or augmented reality headset could lead to better display performance and enable novel methods of user interaction with the system. However, it remains a challenge for a system to combine the required operational speed and accuracy of eye tracking with a technology that has a small enough form factor and weight to be easily integrated into a user-friendly headset. By using small, lightweight hardware comprising a high frame rate microLED array and fast single pixel detector, we report a model eye tracking system based on single pixel tracking and a specially developed set of deep learned illumination patterns. This model system is used to demonstrate eye tracking with an angular accuracy of better than one degree and a measurement rate of up to $3.59 \,$ kHz.

Figures (7)

• Figure 1: (a) Schematic diagram of the experimental setup. Light from the microLED array is coupled into a lens taken from a pair of AR glasses, where the light proceeds through the internal optics until it is projected onto the model eye and collected at the single pixel detector. (b) Photograph of the experimental setup, with blue light coming from the microLED array. The mirror in the upper right of the image is reflecting this light into the AR lens at the top left of the image. This light is then focused onto the model eye ball towards the bottom left of the image. (c) The Vuzik Blade 2™ $\,$ AR glasses and the lens removed from the right eye position that was integrated into the optical setup. (d) A close up photograph of the lens from the AR glasses. Two of the internal optical components of this lens are faintly visible in this image. There is a darker shaded region that is used to direct the light through the lens to the display region. The fainter display region is indicated with a checked line as a guide for the eye in this image. The light patterns are coupled into the lens via the optics on the right hand side of this image. (e) A close up of the model eye illuminated by a Hadamard pattern that has been transmitted through the optical system. The eye is centred in the horizontal plane, but has some declination and therefore the iris appears below the projected pattern
• Figure 2: (a), (b) and (c) show examples of the eye imaged from the Hadamard pattern training data. These show the eye angular position ($\phi$,$\theta$) at; (20,10), (-12,-12) and (35,-19), respectively. (d) Illustrates these positions on the graphic used throughout this paper. Examples of the illumination patterns used in this work are also shown, (e) one of the 480 Hadamard patterns used in this work and (f) one of the 72 generated deep learned patterns.
• Figure 3: Classification results for (a) the deep learned pattern sets and (b) the Hadamard pattern sets. The figure shows the five different categories as various shades of grey. These correspond to different sectors that the eye position could be in and correspond to an eye looking up, down, left, right or approximately straight ahead in the light grey central region. The classification results are shown by the coloured symbols. Each symbol represents a test measurement eye position. 12-15 measurements were taken at each position and if every one of these measurements correctly predicts the region, a blue circle is shown on the diagram at the measurement position. If at least one of those sample measurements is an incorrect prediction an orange diamond is plotted at the measurement position.
• Figure 4: Confusion matrix for the classification results for (a) the deep learned patterns and (b) Hadamard patterns. This shows the classification and misclassification of the five different categories for each pattern set.
• Figure 5: Results of the regression measurements with (a) the deep learned patterns and (b) the Hadamard patterns. The dot colour is related to the root mean squared error (RMSSE) of the position determined for the measurements at each position. To further aid a comparative understanding of the results, the size of the dots increases with increasing RMSE. The dot size scale is the same for both figures.