Training Models to Detect Successive Robot Errors from Human Reactions

TL;DR

The paper addresses detecting successive robot errors from human reactions in HRI. It collects video-based multimodal cues from 26 participants and trains per-participant and cross-participant models using facial, pose, audio, and text features to detect errors and classify error stages. The best results reach about 0.935 accuracy for error detection and 0.841 for successive error classification, illustrating that reaction progression signals can be informative. These findings suggest potential for anticipatory adaptation in HRI systems, though generalization to unseen users warrants further study.

Abstract

As robots become more integrated into society, detecting robot errors is essential for effective human-robot interaction (HRI). When a robot fails repeatedly, how can it know when to change its behavior? Humans naturally respond to robot errors through verbal and nonverbal cues that intensify over successive failures-from confusion and subtle speech changes to visible frustration and impatience. While prior work shows that human reactions can indicate robot failures, few studies examine how these evolving responses reveal successive failures. This research uses machine learning to recognize stages of robot failure from human reactions. In a study with 26 participants interacting with a robot that made repeated conversational errors, behavioral features were extracted from video data to train models for individual users. The best model achieved 93.5% accuracy for detecting errors and 84.1% for classifying successive failures. Modeling the progression of human reactions enhances error detection and understanding of repeated interaction breakdowns in HRI.

Figures (1)

• Figure 1: A vignette of a participant's interaction with a robot (adapted from our earlier work lbr).