Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NeuGaze: Reshaping the future BCI

Yiqian Yang

TL;DR

NeuGaze proposes a webcam-driven BCI substitute that uses eye gaze, head pose, and facial expressions to achieve hands-free control with a standard 30 Hz camera. By integrating Mediapipe-based facial analysis, L2CS-NET gaze estimation, and a LASSO-CV mapper, the system delivers gaze-to-mouse control, thresholded intent detection, and a wheel-based key-selection mechanism to manage 27 game actions. The approach demonstrates real-time interaction in first-person gaming and cursor control, highlighting potential benefits for motor-impaired users by avoiding specialized hardware. While promising, the study is limited to a single user and controlled conditions; future work should pursue multi-user validation, quantitative benchmarking, and broader non-gaming assistive applications to establish generalizability and impact.

Abstract

Traditional brain-computer interfaces (BCIs), reliant on costly electroencephalography or invasive implants, struggle with complex human-computer interactions due to setup complexity and limited precision. We present NeuGaze, a novel webcam-based system that leverages eye gaze, head movements, and facial expressions to enable intuitive, real-time control using only a standard 30 Hz webcam, often pre-installed in laptops. Requiring minimal calibration, NeuGaze achieves performance comparable to conventional inputs, supporting precise cursor navigation, key triggering via an efficient skill wheel, and dynamic gaming interactions, such as defeating formidable opponents in first-person games. By harnessing preserved neck-up functionalities in motor-impaired individuals, NeuGaze eliminates the need for specialized hardware, offering a low-cost, accessible alternative to BCIs. This paradigm empowers diverse applications, from assistive technology to entertainment, redefining human-computer interaction for motor-impaired users. Project is at \href{https://github.com/NeuSpeech/NeuGaze}{github.com/NeuSpeech/NeuGaze}.

NeuGaze: Reshaping the future BCI

TL;DR

NeuGaze proposes a webcam-driven BCI substitute that uses eye gaze, head pose, and facial expressions to achieve hands-free control with a standard 30 Hz camera. By integrating Mediapipe-based facial analysis, L2CS-NET gaze estimation, and a LASSO-CV mapper, the system delivers gaze-to-mouse control, thresholded intent detection, and a wheel-based key-selection mechanism to manage 27 game actions. The approach demonstrates real-time interaction in first-person gaming and cursor control, highlighting potential benefits for motor-impaired users by avoiding specialized hardware. While promising, the study is limited to a single user and controlled conditions; future work should pursue multi-user validation, quantitative benchmarking, and broader non-gaming assistive applications to establish generalizability and impact.

Abstract

Traditional brain-computer interfaces (BCIs), reliant on costly electroencephalography or invasive implants, struggle with complex human-computer interactions due to setup complexity and limited precision. We present NeuGaze, a novel webcam-based system that leverages eye gaze, head movements, and facial expressions to enable intuitive, real-time control using only a standard 30 Hz webcam, often pre-installed in laptops. Requiring minimal calibration, NeuGaze achieves performance comparable to conventional inputs, supporting precise cursor navigation, key triggering via an efficient skill wheel, and dynamic gaming interactions, such as defeating formidable opponents in first-person games. By harnessing preserved neck-up functionalities in motor-impaired individuals, NeuGaze eliminates the need for specialized hardware, offering a low-cost, accessible alternative to BCIs. This paradigm empowers diverse applications, from assistive technology to entertainment, redefining human-computer interaction for motor-impaired users. Project is at \href{https://github.com/NeuSpeech/NeuGaze}{github.com/NeuSpeech/NeuGaze}.

Paper Structure

This paper contains 20 sections, 14 figures.

Table of Contents

  1. Introduction
  2. Invasive Devices
  3. Non-Invasive Devices
  4. Applicable Medical Conditions
  5. Methods
  6. Task
  7. Overview
  8. System Components
  9. Gaze-Based Mouse Control
  10. Intention Detection via Blend Shapes
  11. Head Pose Utilization
  12. Keys mapping
  13. Result
  14. First-Person Gaming Controls
  15. Cursor item selection
  16. ...and 5 more sections

Figures (14)

  • Figure 1: The NeuGaze framework captures a 30 Hz webcam image, processed by Mediapipe for facial landmarks, blend shapes, and head poses. Landmarks define a face box, segmented for the L2CS model to predict gaze angles (yaw, pitch). These, with the face box, feed a LASSOCV regression model to estimate gaze points for mouse control. Blend shapes, thresholded into binary values, combine logically to detect intentions (e.g., jaw movements), mapping to key presses or mouse clicks. Head poses (pitch, yaw, roll) control mouse movement, key selection, or scrolling, with mode switching based on user preference.
  • Figure 2: Changing the camera perspective.
  • Figure 3: Moving forward the character.
  • Figure 5: Converting gaze point coordinates to mouse cursor position.
  • Figure 6: Head movement refines cursor position when gaze tracking is inaccurate.
  • ...and 9 more figures