Gazing at Failure: Investigating Human Gaze in Response to Robot Failure in Collaborative Tasks

TL;DR

The paper investigates how human gaze dynamics respond to two robot failure types (executional and decisional) occurring at different task phases (early or late) in a Tangram collaboration, plus the effect of failure acknowledgement. Using eye-tracking and AoI-based gaze metrics, it shows that executional failures induce more gaze shifts toward the robot, while decisional failures reduce gaze transition entropy among areas of interest, with timing further modulating attention distribution. Subjective measures indicate that executional and early failures can boost perceived intelligence and trust, while late failures can increase perceived safety, and acknowledgement yields mixed effects. The study demonstrates that gaze can serve as a robust indicator of robot failure and type, informing future failure repair and adaptive HRC strategies with practical implications for maintaining trust in shared autonomy systems.

Abstract

Robots are prone to making errors, which can negatively impact their credibility as teammates during collaborative tasks with human users. Detecting and recovering from these failures is crucial for maintaining effective level of trust from users. However, robots may fail without being aware of it. One way to detect such failures could be by analysing humans' non-verbal behaviours and reactions to failures. This study investigates how human gaze dynamics can signal a robot's failure and examines how different types of failures affect people's perception of robot. We conducted a user study with 27 participants collaborating with a robotic mobile manipulator to solve tangram puzzles. The robot was programmed to experience two types of failures -- executional and decisional -- occurring either at the beginning or end of the task, with or without acknowledgement of the failure. Our findings reveal that the type and timing of the robot's failure significantly affect participants' gaze behaviour and perception of the robot. Specifically, executional failures led to more gaze shifts and increased focus on the robot, while decisional failures resulted in lower entropy in gaze transitions among areas of interest, particularly when the failure occurred at the end of the task. These results highlight that gaze can serve as a reliable indicator of robot failures and their types, and could also be used to predict the appropriate recovery actions.

Figures (6)

• Figure 1: Different areas of interest in the experiment.
• Figure 2: Experimental diagram showing the process where participants first complete a pre-task questionnaire, followed by collaboratively solving a Tangram puzzle four times.
• Figure 3: The average number of gaze shifts across all AoIs (left) and toward the robot body (right) across three different failure situations, with or without the robot acknowledging its failure. Error bars represent the standard error of the mean. Significance levels, based on adjusted p-values, are denoted as follows: ** for $p < .01$, and **** for $p < .0001$.
• Figure 4: The average number of gaze shifts across all AoIs and the average number of gaze shifts toward the robot body, comparing failure type, failure timing, and the robot’s acknowledgement of its failure. Error bars represent the standard error of the mean. Significance levels are denoted as follows: *** for $p < .001$.
• Figure 5: The average proportion of participant gazes directed at the end effector (EE), robot body/face (RB), and Tangram figure (T) during puzzle solving, across three different failure situations, with or without the robot acknowledging its failure. Error bars represent the standard error of the mean. Significance levels, based on adjusted p-values, are denoted as follows: * for $p < .05$, ** for $p < .01$, and *** for $p < .001$.