Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robot Error Awareness Through Human Reactions: Implementation, Evaluation, and Recommendations

Maia Stiber, Russell Taylor, Chien-Ming Huang

TL;DR

This paper addresses real-time error detection in human-robot collaboration by leveraging implicit social signals (facial action units and speech) alongside explicit user reports. It introduces a proactive, two-phase implicit detection framework that uses contextual robot status and verification queries to reduce false positives, and integrates explicit error reports for robustness. In a between-subject study (N = 28), the proactive system detections were faster and more favorably perceived than a reactive baseline, across assembly and packing tasks, with evidence of increased user engagement and satisfaction. The authors provide practical recommendations for enabling robot error awareness and outline limitations and directions for future work, including continual learning and longer-term evaluations.

Abstract

Effective error detection is crucial to prevent task disruption and maintain user trust. Traditional methods often rely on task-specific models or user reporting, which can be inflexible or slow. Recent research suggests social signals, naturally exhibited by users in response to robot errors, can enable more flexible, timely error detection. However, most studies rely on post hoc analysis, leaving their real-time effectiveness uncertain and lacking user-centric evaluation. In this work, we developed a proactive error detection system that combines user behavioral signals (facial action units and speech), user feedback, and error context for automatic error detection. In a study (N = 28), we compared our proactive system to a status quo reactive approach. Results show our system 1) reliably and flexibly detects error, 2) detects errors faster than the reactive approach, and 3) is perceived more favorably by users than the reactive one. We discuss recommendations for enabling robot error awareness in future HRI systems.

Robot Error Awareness Through Human Reactions: Implementation, Evaluation, and Recommendations

TL;DR

This paper addresses real-time error detection in human-robot collaboration by leveraging implicit social signals (facial action units and speech) alongside explicit user reports. It introduces a proactive, two-phase implicit detection framework that uses contextual robot status and verification queries to reduce false positives, and integrates explicit error reports for robustness. In a between-subject study (N = 28), the proactive system detections were faster and more favorably perceived than a reactive baseline, across assembly and packing tasks, with evidence of increased user engagement and satisfaction. The authors provide practical recommendations for enabling robot error awareness and outline limitations and directions for future work, including continual learning and longer-term evaluations.

Abstract

Effective error detection is crucial to prevent task disruption and maintain user trust. Traditional methods often rely on task-specific models or user reporting, which can be inflexible or slow. Recent research suggests social signals, naturally exhibited by users in response to robot errors, can enable more flexible, timely error detection. However, most studies rely on post hoc analysis, leaving their real-time effectiveness uncertain and lacking user-centric evaluation. In this work, we developed a proactive error detection system that combines user behavioral signals (facial action units and speech), user feedback, and error context for automatic error detection. In a study (N = 28), we compared our proactive system to a status quo reactive approach. Results show our system 1) reliably and flexibly detects error, 2) detects errors faster than the reactive approach, and 3) is perceived more favorably by users than the reactive one. We discuss recommendations for enabling robot error awareness in future HRI systems.
Paper Structure (40 sections, 4 figures)

This paper contains 40 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Status Quo Techniques for Error Detection in HRI
  4. Role of Social Signals and Error Detection
  5. Proactivity in Error Management
  6. Proactive Robot Error Detection System
  7. System Overview
  8. Perception
  9. Intent Recognition
  10. Implicit Error Detection
  11. Phase 1---Initial Potential Error Detection
  12. Phase 2---Verification
  13. Explicit Error Detection
  14. Error Mitigation and Recovery
  15. Robot Controller
  16. ...and 25 more sections

Figures (4)

  • Figure 1: System diagram with a zoomed-in view of the implicit error detection component to highlight the two-phased detection process based on the implicit indicator modality.
  • Figure 2: Evaluation study setup for assembly and packing tasks.
  • Figure 3: Assembly Results: bars depict mean; error bars are standard error. (a) Impact of system type on detection timeliness (error detection delay). (b) Detection methods' impact on error detection delay for the proactive system. (c) Impact of system type on error detection delay for explicit detection. Participants' perception of systems' error handling and robot. Cross is mean; box shows quartiles; line shows median. (d) Impact of system type on satisfaction with error handling. (e) Impact of system type on perception of robot as a teammate.
  • Figure 4: Packing Results: bars depict mean; error bars are standard error. (a) Impact of system type on detection timeliness (error detection delay). (b) Detection methods' impact on error detection delay for the proactive system. (c) Impact of system type on error detection delay for explicit detection.