Co-designing a Sub-millisecond Latency Event-based Eye Tracking System with Submanifold Sparse CNN

TL;DR

This work presents SEE, a hardware-software co-design for sub-millisecond event-based eye tracking that leverages a submanifold sparse CNN backbone implemented on an FPGA accelerator and a GRU+FC head on an ARM host. By exploiting input sparsity and a specialized Sparse Dataflow, SEE achieves ultra-low latency with high p10 accuracy on the AIS2024 dataset, outperforming embedded GPUs by large margins. The paper introduces a co-optimization framework that searches for compact architectures balancing accuracy and hardware latency, and demonstrates substantial energy efficiency (low mJ per inference) alongside strong performance. The approach enables practical, real-time eye tracking for VR/AR and similar applications, with potential for further improvements by integrating recurrent/attention modules into the FPGA dataflow and refining quantization techniques.

Abstract

Eye-tracking technology is integral to numerous consumer electronics applications, particularly in the realm of virtual and augmented reality (VR/AR). These applications demand solutions that excel in three crucial aspects: low-latency, low-power consumption, and precision. Yet, achieving optimal performance across all these fronts presents a formidable challenge, necessitating a balance between sophisticated algorithms and efficient backend hardware implementations. In this study, we tackle this challenge through a synergistic software/hardware co-design of the system with an event camera. Leveraging the inherent sparsity of event-based input data, we integrate a novel sparse FPGA dataflow accelerator customized for submanifold sparse convolution neural networks (SCNN). The SCNN implemented on the accelerator can efficiently extract the embedding feature vector from each representation of event slices by only processing the non-zero activations. Subsequently, these vectors undergo further processing by a gated recurrent unit (GRU) and a fully connected layer on the host CPU to generate the eye centers. Deployment and evaluation of our system reveal outstanding performance metrics. On the Event-based Eye-Tracking-AIS2024 dataset, our system achieves 81% p5 accuracy, 99.5% p10 accuracy, and 3.71 Mean Euclidean Distance with 0.7 ms latency while only consuming 2.29 mJ per inference. Notably, our solution opens up opportunities for future eye-tracking systems. Code is available at https://github.com/CASR-HKU/ESDA/tree/eye_tracking.

Figures (6)

• Figure 1: Software architecture: For an event stream, we partition it into multiple consecutive clips. These clips are then transformed into sparse voxel representations. Subsequently, an SCNN is used to generate feature embeddings, which will be then fed into a Gated Recurrent Unit (GRU) module. The GRU module generates the hidden state, and a Fully Connected (FC) layer regresses the eye centers.
• Figure 2: Standard and submanifold sparse convolution. For standard convolution, all the pixels in an image are processed by the kernels equally, leading to the dilation of spatial intensity. On the other hand, submanifold sparse convolution ensures that the output non-zero pixels locations to be identical as the input.
• Figure 3: Heterogeneous Hardware architecture. The proposed hardware system primarily consists of the Arm Cortex-A53 acting as a processing system and the SCNN accelerator implemented on programable logic. The input to the SCNN accelerator is the sparse features and a binary bitmap to record the non-zero pixel locations. The GRU and fully connected layers are executed in the processing system with Arm NEON SIMD (Single Instruction, Multiple Data) engine.
• Figure 4: Network searching pipeline: We sample networks from a search pool and use a hardware simulator to select low-latency ones. After training these networks, we create a latency-accuracy Pareto frontier to show the trade-off between accuracy and latency.
• Figure 5: Accuracy vs Latency for sampled models. (a) p5 Accuracy vs. Latency. (b) p10 Accuracy vs. Latency. (c) Mean Euclidean Distances vs. Latency