Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Continual Learning through Human-Robot Interaction: Human Perceptions of a Continual Learning Robot in Repeated Interactions

Ali Ayub, Zachary De Francesco, Patrick Holthaus, Chrystopher L. Nehaniv, Kerstin Dautenhahn

TL;DR

This study investigates how humans perceive robots that continually learn from their interactions over multiple sessions. It presents a socially guided continual learning (SGCL) framework implemented on a Fetch mobile manipulator, comparing Finetuning (FT), centroid-based concept learning (CBCL), and joint training (JT) across five teaching/testing sessions with 60 participants. Findings show that forgetting learned objects degrades trust, competence, and usability, while task load remains stable; CBCL and JT generally outperform FT in user-perceived attributes, though real-world CL performance remains imperfect (≈45–60% accuracy) when learning from humans. The results suggest that effectiveness in long-term HRI hinges on learning reliability and user-centered design, and call for integrating interpretability and user guidance to reduce labeling ambiguities. Overall, the work demonstrates the feasibility of personalized CL robots while highlighting the need for improved real-world robustness and human-centric training signals.

Abstract

For long-term deployment in dynamic real-world environments, assistive robots must continue to learn and adapt to their environments. Researchers have developed various computational models for continual learning (CL) that can allow robots to continually learn from limited training data, and avoid forgetting previous knowledge. While these CL models can mitigate forgetting on static, systematically collected datasets, it is unclear how human users might perceive a robot that continually learns over multiple interactions with them. In this paper, we developed a system that integrates CL models for object recognition with a Fetch mobile manipulator robot and allows human participants to directly teach and test the robot over multiple sessions. We conducted an in-person study with 60 participants that interacted with our system in 300 sessions (5 sessions per participant). We conducted a between-subject study with three different CL models to understand human perceptions of continual learning robots over multiple sessions. Our results suggest that participants' perceptions of trust, competence, and usability of a continual learning robot significantly decrease over multiple sessions if the robot forgets previously learned objects. However, the perceived task load on participants for teaching and testing the robot remains the same over multiple sessions even if the robot forgets previously learned objects. Our results also indicate that state-of-the-art CL models might perform unreliably when applied on robots interacting with human participants. Further, continual learning robots are not perceived as very trustworthy or competent by human participants, regardless of the underlying continual learning model or the session number.

Continual Learning through Human-Robot Interaction: Human Perceptions of a Continual Learning Robot in Repeated Interactions

TL;DR

This study investigates how humans perceive robots that continually learn from their interactions over multiple sessions. It presents a socially guided continual learning (SGCL) framework implemented on a Fetch mobile manipulator, comparing Finetuning (FT), centroid-based concept learning (CBCL), and joint training (JT) across five teaching/testing sessions with 60 participants. Findings show that forgetting learned objects degrades trust, competence, and usability, while task load remains stable; CBCL and JT generally outperform FT in user-perceived attributes, though real-world CL performance remains imperfect (≈45–60% accuracy) when learning from humans. The results suggest that effectiveness in long-term HRI hinges on learning reliability and user-centered design, and call for integrating interpretability and user guidance to reduce labeling ambiguities. Overall, the work demonstrates the feasibility of personalized CL robots while highlighting the need for improved real-world robustness and human-centric training signals.

Abstract

For long-term deployment in dynamic real-world environments, assistive robots must continue to learn and adapt to their environments. Researchers have developed various computational models for continual learning (CL) that can allow robots to continually learn from limited training data, and avoid forgetting previous knowledge. While these CL models can mitigate forgetting on static, systematically collected datasets, it is unclear how human users might perceive a robot that continually learns over multiple interactions with them. In this paper, we developed a system that integrates CL models for object recognition with a Fetch mobile manipulator robot and allows human participants to directly teach and test the robot over multiple sessions. We conducted an in-person study with 60 participants that interacted with our system in 300 sessions (5 sessions per participant). We conducted a between-subject study with three different CL models to understand human perceptions of continual learning robots over multiple sessions. Our results suggest that participants' perceptions of trust, competence, and usability of a continual learning robot significantly decrease over multiple sessions if the robot forgets previously learned objects. However, the perceived task load on participants for teaching and testing the robot remains the same over multiple sessions even if the robot forgets previously learned objects. Our results also indicate that state-of-the-art CL models might perform unreliably when applied on robots interacting with human participants. Further, continual learning robots are not perceived as very trustworthy or competent by human participants, regardless of the underlying continual learning model or the session number.
Paper Structure (23 sections, 1 equation, 8 figures, 5 tables)

This paper contains 23 sections, 1 equation, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Socially Guided Continual Learning
  4. Continual Learning Models
  5. Method
  6. Hypotheses
  7. Fetch Mobile Manipulator Robot
  8. Graphical User Interface
  9. Participants
  10. Procedure
  11. Measures
  12. Results
  13. Classification Accuracy
  14. Cognition based trust
  15. Robot social attributes
  16. ...and 8 more sections

Figures (8)

  • Figure 1: (Left) Experimental layout for the SGCL setup with the participant and the robot. (Right) Corresponding real-world setup.
  • Figure 2: Our complete SGCL system. Processed RGB images from robot's camera are sent to the GUI for transparency and also passed on to the CL Model. The user sends object names to the CL model either for training the CL model or finding an object. The arm trajectory planner takes point cloud data, processed RGB data, and predicted object labels from the CL model as input and sends the arm trajectory for the Fetch robot to point to the object.
  • Figure 3: The graphical user interface (GUI) used to interact with the robot. The RGB camera output with bounding boxes is on the top left. The buttons at the bottom can be used to teach objects to the robot and ask it to find objects in the testing phase. The top right of the GUI shows information sent by the robot to the user.
  • Figure 4: Average classification accuracy of the three CL models over 5 sessions. The dotted line at the bottom represents the random chance of predicting a correct object in each session. The shaded areas represent the standard deviation for CL models.
  • Figure 5: Boxplots for cognition based trust scores on the HCT scale, ranging from 1 to 5. Significance levels ($* := p < .05; ** := p < 0.01; *** :=p < 0.001$) are indicated on bars between the columns.
  • ...and 3 more figures