Real-Time Detection of Robot Failures Using Gaze Dynamics in Collaborative Tasks

TL;DR

This work demonstrates that human gaze dynamics can be leveraged to detect robot failures in real-time during collaborative Tangram tasks. By extracting gaze features and training multiple classifiers, the study shows high accuracy for distinguishing executional failures (≈90%) and decisional failures (≈80%) within the first 5 seconds of failure, using leave-one-out cross-validation on 26 participants. In real-time scenarios, random forest offers the best overall performance (~60% accuracy) across 3–10 second windows, with SVM providing higher failure recall. The findings highlight the potential of gaze-based failure detection to enhance trust and robustness in human-robot collaboration, while acknowledging the benefits of incorporating multimodal cues in future work.

Abstract

Detecting robot failures during collaborative tasks is crucial for maintaining trust in human-robot interactions. This study investigates user gaze behaviour as an indicator of robot failures, utilising machine learning models to distinguish between non-failure and two types of failures: executional and decisional. Eye-tracking data were collected from 26 participants collaborating with a robot on Tangram puzzle-solving tasks. Gaze metrics, such as average gaze shift rates and the probability of gazing at specific areas of interest, were used to train machine learning classifiers, including Random Forest, AdaBoost, XGBoost, SVM, and CatBoost. The results show that Random Forest achieved 90% accuracy for detecting executional failures and 80% for decisional failures using the first 5 seconds of failure data. Real-time failure detection was evaluated by segmenting gaze data into intervals of 3, 5, and 10 seconds. These findings highlight the potential of gaze dynamics for real-time error detection in human-robot collaboration.

Figures (3)

• Figure 1: Classifier accuracy assessed using the first n seconds of the failure period: a) distinguishing between non-failure and executional failure, and b) distinguishing between non-failure and decisional failure.
• Figure 2: Classifier performance evaluated using eye-tracking metrics segmented into intervals of 3, 5, and 10 seconds with a 1-second sliding window: a) Accuracy in distinguishing between non-failure and executional failure, b) Accuracy in distinguishing between non-failure and decisional failure, c) Recall in detecting executional failure, and d) Recall in detecting decisional failure.
• Figure 3: Percentage of users for whom the model successfully detected failures during each 5-second interval of the failure phase: a) Executional Failure, and b) Decisional Failure.