Eye-Tracking-Driven Control in Daily Task Assistance for Assistive Robotic Arms

TL;DR

This work tackles robust gaze-based control for daily-life assistance using assistive robotic arms by relocating object localization to an eye-in-hand camera and employing task pictograms as fiducials. The system integrates YOLOv12n for detection and a FLANN-AKAZE-based feature-matching pipeline to transmit object data between the eye-tracking setup and the robot, reducing reliance on precise 3D gaze estimates. Results show high task-selection reliability (up to 97.9% in measurements) and low latency (about 260 ms), with a modular, open-source framework that supports rapid adaptation to new tasks and objects. The study provides practical insights, including algorithm trade-offs and fallback mechanisms, and outlines directions for future real-world validation and multimodal enhancements to improve independence and safety for users with severe disabilities.

Abstract

Shared control improves Human-Robot Interaction by reducing the user's workload and increasing the robot's autonomy. It allows robots to perform tasks under the user's supervision. Current eye-tracking-driven approaches face several challenges. These include accuracy issues in 3D gaze estimation and difficulty interpreting gaze when differentiating between multiple tasks. We present an eye-tracking-driven control framework, aimed at enabling individuals with severe physical disabilities to perform daily tasks independently. Our system uses task pictograms as fiducial markers combined with a feature matching approach that transmits data of the selected object to accomplish necessary task related measurements with an eye-in-hand configuration. This eye-tracking control does not require knowledge of the user's position in relation to the object. The framework correctly interpreted object and task selection in up to 97.9% of measurements. Issues were found in the evaluation, that were improved and shared as lessons learned. The open-source framework can be adapted to new tasks and objects due to the integration of state-of-the-art object detection models.

Figures (6)

• Figure 1: The computation process of the framework. The object is selected by gaze. The pictogram and the object are detected in both camera scenes and matched using a feature-matching approach. Coordinate transformation provides an estimated location of the object for further interaction.
• Figure 2: FM test cases. Upper scene: Inside scene of a set table. Lower scene: Outside scene of a set table in direct sunlight.
• Figure 3: Feature matching results for selected algorithms. In the best cases, shown on the right 32 (BF-ORB), 87 (FLANN-AKAZE), and 53 (FLANN-SIFT) features were matched and not removed by the ratio test. In the worst cases, shown on the left 2 (BF-ORB), 3 (FLANN-AKAZE), and 12 (FLANN-SIFT) features were matched. Color changes in contrast to Figure \ref{['fig:fig_testcases']} result from using OpenCV library and have no impact on performance.
• Figure 4: Program flow for robust selection information transmission between eye-tracker and robot field of view.
• Figure 5: The framework of the eye-tracking controller. The depiction shows communication between nodes based on the ROS2 RQT graph. The rectangles represent nodes and the circles represent topics with message types. The green rectangles indicate interfaces to the periphery.