Helios 2.0: A Robust, Ultra-Low Power Gesture Recognition System Optimised for Event-Sensor based Wearables

TL;DR

Helios 2.0 demonstrates an ultra-low-power, event-based gesture recognition system for smart glasses, achieving state-of-the-art F1 performance while operating at 6–8 mW on a DSP. By leveraging polarity-separated time surfaces, longer 2 s sequences, and a five-stage architecture with quantisation-aware training, it surpasses prior results and enables real-time, touch-free interaction in varied environments. The work combines synthetic data generation, comprehensive real-world benchmarks, and ablation studies to show robustness to human variability and environmental changes, while maintaining low latency (≈$2.34$ ms). This approach significantly advances wearable vision by delivering accurate, energy-efficient gesture control suitable for next-generation smart glasses and broader event-based sensing applications.

Abstract

We present an advance in wearable technology: a mobile-optimized, real-time, ultra-low-power event camera system that enables natural hand gesture control for smart glasses, dramatically improving user experience. While hand gesture recognition in computer vision has advanced significantly, critical challenges remain in creating systems that are intuitive, adaptable across diverse users and environments, and energy-efficient enough for practical wearable applications. Our approach tackles these challenges through carefully selected microgestures: lateral thumb swipes across the index finger (in both directions) and a double pinch between thumb and index fingertips. These human-centered interactions leverage natural hand movements, ensuring intuitive usability without requiring users to learn complex command sequences. To overcome variability in users and environments, we developed a novel simulation methodology that enables comprehensive domain sampling without extensive real-world data collection. Our power-optimised architecture maintains exceptional performance, achieving F1 scores above 80\% on benchmark datasets featuring diverse users and environments. The resulting models operate at just 6-8 mW when exploiting the Qualcomm Snapdragon Hexagon DSP, with our 2-channel implementation exceeding 70\% F1 accuracy and our 6-channel model surpassing 80\% F1 accuracy across all gesture classes in user studies. These results were achieved using only synthetic training data. This improves on the state-of-the-art for F1 accuracy by 20\% with a power reduction 25x when using DSP. This advancement brings deploying ultra-low-power vision systems in wearable devices closer and opens new possibilities for seamless human-computer interaction.

Figures (17)

• Figure 1: Our system enables natural hand gesture interaction with smart glasses without requiring additional peripherals. The event-based vision approach allows for low-power, responsive gesture recognition even in challenging lighting conditions.
• Figure 2: Hardware used for data collection and model testing. The event-camera is mounted on the left side of the glasses, with the display connection on the right.
• Figure 3: Examples of the 3D hand models used for simulation.
• Figure 4: Examples of the 3D environments used within simulation.
• Figure 5: Left to right, pinch, left swipe, right swipe, with the end point of the gesture illustrated on the bottom.