Realizing Fully-Integrated, Low-Power, Event-Based Pupil Tracking with Neuromorphic Hardware

TL;DR

This work tackles the challenge of real-time, low-power pupil tracking on wearable devices by presenting a fully integrated neuromorphic solution that combines event-based sensing with on-device processing on two Speck2f chips and an MCU for decoding. The authors design a hardware-aware SNN with uncertainty-aware, gated temporal decoding and a training-and-deployment pipeline that respects Speck2f constraints, achieving 100 Hz binocular tracking at under $5$ mW per eye in a wearable prototype. Key contributions include a novel uncertainty-quantifying SNN, a tight integration strategy leveraging SPI-based readout and a cyclic readout scheme, and comprehensive validation on a real-world multi-user dataset plus wearable demonstrations showing robust performance and energy efficiency. The results illustrate the practicality of end-to-end neuromorphic computing for always-on eye tracking and set a blueprint for energy-efficient, event-based vision in future wearables and related domains.

Abstract

Eye tracking is fundamental to numerous applications, yet achieving robust, high-frequency tracking with ultra-low power consumption remains challenging for wearable platforms. While event-based vision sensors offer microsecond resolution and sparse data streams, they have lacked fully integrated, low-power processing solutions capable of real-time inference. In this work, we present the first battery-powered, wearable pupil-center-tracking system with complete on-device integration, combining event-based sensing and neuromorphic processing on the commercially available Speck2f system-on-chip with lightweight coordinate decoding on a low-power microcontroller. Our solution features a novel uncertainty-quantifying spiking neural network with gated temporal decoding, optimized for strict memory and bandwidth constraints, complemented by systematic deployment mechanisms that bridge the reality gap. We validate our system on a new multi-user dataset and demonstrate a wearable prototype with dual neuromorphic devices achieving robust binocular pupil tracking at 100 Hz with an average power consumption below 5 mW per eye. Our work demonstrates that end-to-end neuromorphic computing enables practical, always-on eye tracking for next-generation energy-efficient wearable systems.

Figures (14)

• Figure 1: Top: Prototype of our fully-integrated, battery-powered pupil-center-tracking system based on neuromorphic technology, featuring two Speck2f devices, IR LEDs, and an nRF52840 MCU. Bottom: Our on-chip SNN processes input events asynchronously, while an MCU runs our off-chip decoding at 100 Hz to yield pupil-center and uncertainty estimates.
• Figure 2: Speck2f support boards used in this work. Left: Speck2f development kit, featuring a Xilinx Artix-7 FPGA. Right: our custom PCB designed to accommodate two Speck2f units and featuring an nRF52840 MCU. The proportions of the boards are maintained to provide an accurate comparison of their sizes.
• Figure 3: Cyclic readout strategy for a four-neuron layer. The MCU sequentially samples each neuron via SPI, synchronized to the SCLK signal, as rapidly as possible. Vertical black lines indicate spike events; colored regions show sampling windows. Dashed arrows mark the completion of each readout cycle, triggering the off-chip decoding process.
• Figure 4: Representative samples from our real-world dataset (left and bottom) show event streams from different users and acquisition conditions. Colors used to distinguish event polarities: red for negative, green for positive. The central heatmap displays the distribution of ground-truth pupil centers, with marginal histograms indicating coverage along each axis.
• Figure 5: Example sequence from our real-world dataset and on-chip model evaluation. Top: Event frames at key time points (white: ground truth; blue: prediction). Middle: Predicted pupil coordinates with uncertainty, event counts, and spikes per layer. Bottom: Processing power consumption for each Speck2f subsystem, with a total average of 4.22 mW. Vertical lines indicate correspondence between frames and time-series data. Results obtained with the Speck2f development kit.