MIHRaGe: A Mixed-Reality Interface for Human-Robot Interaction via Gaze-Oriented Control

TL;DR

This paper addresses accessibility for individuals with upper-limb impairments by presenting MIHRAGe, a gaze-driven interface that fuses eye-tracking, a mixed-reality projection, and a six-DOF robot arm for hands-free manipulation. The approach provides real-time visual feedback of gaze selections and robot intent, linking interface gaze to robot actions via a marker-based alignment and a linear coordinate transform. Experimental evaluation with four participants showed gaze accuracy around 1.46 cm and robot positioning errors around 1.5 cm, achieving a 95% success rate for pick actions and 65% for placing in a pick-and-place task, highlighting both the potential and current limitations (e.g., depth perception and contact dynamics). The results suggest that gaze-based control with immersive feedback can meaningfully improve accessibility in human-robot interaction, with future work focusing on calibration, depth-aware perception, haptic feedback, and broader user testing to extend real-world applicability.

Abstract

Individuals with upper limb mobility impairments often require assistive technologies to perform activities of daily living. While gaze-tracking has emerged as a promising method for robotic assistance, existing solutions lack sufficient feedback mechanisms, leading to uncertainty in user intent recognition and reduced adaptability. This paper presents the MIHRAGe interface, an integrated system that combines gaze-tracking, robotic assistance, and a mixed-reality to create an immersive environment for controlling the robot using only eye movements. The system was evaluated through an experimental protocol involving four participants, assessing gaze accuracy, robotic positioning precision, and the overall success of a pick and place task. Results showed an average gaze fixation error of 1.46 cm, with individual variations ranging from 1.28 cm to 2.14 cm. The robotic arm demonstrated an average positioning error of +-1.53 cm, with discrepancies attributed to interface resolution and calibration constraints. In a pick and place task, the system achieved a success rate of 80%, highlighting its potential for improving accessibility in human-robot interaction with visual feedback to the user.

Figures (6)

• Figure 1: Overview of the MIHRAGe interface: a user equipped with an eye tracker controls a robotic arm through a mixed-reality interface, which shows user control and robot intent.
• Figure 2: An overview of the interface used to create the mixed-reality environment and the main frames. The interface frame is represented by $\Sigma'$, the robot frame by $\Sigma_r$, and the user frame/eye tracker's frame by $\Sigma$. The workspace for interaction is placed between the four markers (numbered from 0 to 3), along with the menu displayed with the option to choose PICK, MOVE, and CANCEL intention. The dashed lines, the frame orientation axis, and the marker numbering were included for illustrative purposes only.
• Figure 3: State machine representing the different behaviors adopted by the robot, depending on the choice (CH) value the user selects. TP stands for Target Point, GP is the Gripper, and FP corresponds to Final Point.
• Figure 4: Experiment for gaze accuracy and robot position evaluation. The user fixes their gaze on specific points on the projection surface and selects MOVE to pass the target to the robot. The robot then moves to the surface and lowers the tool, enabling the measurement.
• Figure 5: Test results for evaluating gaze fixation accuracy on the a) interface and b) robot. Each square in the heatmap represents a fixed point on the projected surface, displaying the average distance from the fixed point and its standard deviation. The results are also shown for data distribution as box plots for the c) interface and d) robot.