Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GazeGrasp: DNN-Driven Robotic Grasping with Wearable Eye-Gaze Interface

Issatay Tokmurziyev, Miguel Altamirano Cabrera, Luis Moreno, Muhammad Haris Khan, Dzmitry Tsetserukou

TL;DR

GazeGrasp addresses the need for hands-free control of collaborative robots by enabling object manipulation via gaze alone. It integrates an ESP32-CAM eye tracker, MediaPipe gaze estimation, and YOLOv8 object detection with a UR10 arm, employing polynomial calibration with $d=3$, Kalman filtering for smooth gaze, and a magnetic-snapping mechanism to enhance precision. The system maps gaze-driven selections to real-world robot coordinates through camera-to-base transformations and allows picking and placing with 3-second gaze commands, demonstrated on a range of objects. Experimental results from 13 participants show a significant 31% reduction in gaze-alignment time when magnetic snapping is enabled, underscoring the approach's potential to elevate accessibility and autonomy in assistive robotics; the work lays a foundation for broader adoption in rehabilitation, healthcare, and industrial automation, with future work focusing on long-term usability and obstacle-rich environments.

Abstract

We present GazeGrasp, a gaze-based manipulation system enabling individuals with motor impairments to control collaborative robots using eye-gaze. The system employs an ESP32 CAM for eye tracking, MediaPipe for gaze detection, and YOLOv8 for object localization, integrated with a Universal Robot UR10 for manipulation tasks. After user-specific calibration, the system allows intuitive object selection with a magnetic snapping effect and robot control via eye gestures. Experimental evaluation involving 13 participants demonstrated that the magnetic snapping effect significantly reduced gaze alignment time, improving task efficiency by 31%. GazeGrasp provides a robust, hands-free interface for assistive robotics, enhancing accessibility and autonomy for users.

GazeGrasp: DNN-Driven Robotic Grasping with Wearable Eye-Gaze Interface

TL;DR

GazeGrasp addresses the need for hands-free control of collaborative robots by enabling object manipulation via gaze alone. It integrates an ESP32-CAM eye tracker, MediaPipe gaze estimation, and YOLOv8 object detection with a UR10 arm, employing polynomial calibration with , Kalman filtering for smooth gaze, and a magnetic-snapping mechanism to enhance precision. The system maps gaze-driven selections to real-world robot coordinates through camera-to-base transformations and allows picking and placing with 3-second gaze commands, demonstrated on a range of objects. Experimental results from 13 participants show a significant 31% reduction in gaze-alignment time when magnetic snapping is enabled, underscoring the approach's potential to elevate accessibility and autonomy in assistive robotics; the work lays a foundation for broader adoption in rehabilitation, healthcare, and industrial automation, with future work focusing on long-term usability and obstacle-rich environments.

Abstract

We present GazeGrasp, a gaze-based manipulation system enabling individuals with motor impairments to control collaborative robots using eye-gaze. The system employs an ESP32 CAM for eye tracking, MediaPipe for gaze detection, and YOLOv8 for object localization, integrated with a Universal Robot UR10 for manipulation tasks. After user-specific calibration, the system allows intuitive object selection with a magnetic snapping effect and robot control via eye gestures. Experimental evaluation involving 13 participants demonstrated that the magnetic snapping effect significantly reduced gaze alignment time, improving task efficiency by 31%. GazeGrasp provides a robust, hands-free interface for assistive robotics, enhancing accessibility and autonomy for users.
Paper Structure (13 sections, 6 equations, 5 figures)

This paper contains 13 sections, 6 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Gaze-Based Robotic Control
  4. Collaborative Robot Interfaces
  5. Remote Manipulation Systems
  6. System Overview
  7. Calibration via Polynomial Regression
  8. Real-Time Gaze Tracking with Kalman Filters
  9. Object Detection and Magnetic Snapping
  10. Transformation from Camera to Robot Base Coordinates
  11. Picking and Placing Objects
  12. Experimental Evaluation
  13. Conclusions and Future Work

Figures (5)

  • Figure 1: GazeGrasp interface for controlling the collaborative robot UR10 using a gaze-based tracking system: (a) The calibration stage for the users. (b) The workspace view with highlighted objects. (c) Experimental setup for the evaluation of gaze-based control.
  • Figure 2: The flow of the task execution: (1) User aims at the object. (2) UR10 goes to the exact position and picks the object. (3) UR10 returns to the initial position. (4) User chooses a new position to put the object.
  • Figure 3: System Architecture of GazeGrasp: The system integrates gaze-tracking, object detection, and robotic control to enable intuitive manipulation through gaze-based interaction.
  • Figure 4: Glasses equipped with an ESP32 CAM. The video is transmitted from the server, allowing remote control.
  • Figure 5: Box plot showing the time required for users to align their gaze with the center of the object under two conditions: with magnetic assistance (ON) and without magnetic assistance (OFF).