Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Power of Combined Modalities in Interactive Robot Learning

Helen Beierling, Anna-Lisa Vollmer

TL;DR

The paper addresses how to teach robots through human-in-the-loop feedback by combining multiple feedback modalities. It uses probabilistic movement primitives ($ProMP$) to represent trajectories and the PIBB$^2$ algorithm to learn a distribution over $ProMP$ weights from multimodal feedback, including six meta-modalities: Guidance, Correction, Demonstrations, Exploration, Speed, and Fallback. In a minigolf task with a Kinova robot, a two-group study shows that combining modalities improves learning performance and user satisfaction, with Group 2 achieving more hits and higher SUS scores (e.g., first hit after $17.56$ vs $34.60$ attempts; $p=0.0007$ for total hits). The results indicate that Guidance and Speed are most valued, while Exploration requires clearer semantics to consistently aid learning, underscoring the practical benefits and design considerations for multimodal interactive robot learning.

Abstract

This study contributes to the evolving field of robot learning in interaction with humans, examining the impact of diverse input modalities on learning outcomes. It introduces the concept of "meta-modalities" which encapsulate additional forms of feedback beyond the traditional preference and scalar feedback mechanisms. Unlike prior research that focused on individual meta-modalities, this work evaluates their combined effect on learning outcomes. Through a study with human participants, we explore user preferences for these modalities and their impact on robot learning performance. Our findings reveal that while individual modalities are perceived differently, their combination significantly improves learning behavior and usability. This research not only provides valuable insights into the optimization of human-robot interactive task learning but also opens new avenues for enhancing the interactive freedom and scaffolding capabilities provided to users in such settings.

The Power of Combined Modalities in Interactive Robot Learning

TL;DR

The paper addresses how to teach robots through human-in-the-loop feedback by combining multiple feedback modalities. It uses probabilistic movement primitives () to represent trajectories and the PIBB algorithm to learn a distribution over weights from multimodal feedback, including six meta-modalities: Guidance, Correction, Demonstrations, Exploration, Speed, and Fallback. In a minigolf task with a Kinova robot, a two-group study shows that combining modalities improves learning performance and user satisfaction, with Group 2 achieving more hits and higher SUS scores (e.g., first hit after vs attempts; for total hits). The results indicate that Guidance and Speed are most valued, while Exploration requires clearer semantics to consistently aid learning, underscoring the practical benefits and design considerations for multimodal interactive robot learning.

Abstract

This study contributes to the evolving field of robot learning in interaction with humans, examining the impact of diverse input modalities on learning outcomes. It introduces the concept of "meta-modalities" which encapsulate additional forms of feedback beyond the traditional preference and scalar feedback mechanisms. Unlike prior research that focused on individual meta-modalities, this work evaluates their combined effect on learning outcomes. Through a study with human participants, we explore user preferences for these modalities and their impact on robot learning performance. Our findings reveal that while individual modalities are perceived differently, their combination significantly improves learning behavior and usability. This research not only provides valuable insights into the optimization of human-robot interactive task learning but also opens new avenues for enhancing the interactive freedom and scaffolding capabilities provided to users in such settings.
Paper Structure (21 sections, 2 equations, 6 figures)

This paper contains 21 sections, 2 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Implementation
  5. Modalities
  6. Base Modality Preference
  7. Meta Modalities
  8. Guidance
  9. Correction
  10. Demonstrations
  11. Exploration
  12. Speed
  13. Fallback
  14. Study Design
  15. Hypotheses
  16. ...and 6 more sections

Figures (6)

  • Figure 1: Study setup. Red depicts the used robot platform, green the task relevant objects, purple the camera for study recording, and yellow the objects the user gets in contact with.
  • Figure 2: This Figure showcases the interface for Group 2, equipped with all meta-modalities and the base modality.
  • Figure 3: This graph illustrates the percentage of successful hits per trial across all participants.
  • Figure 4: This diagram showcases the ranking distribution of the meta modalities. It illustrates the percentage placement of each meta modality across the ranking positions by Group 2 participants.
  • Figure 5: This Figure illustrates, on one hand, the relative usage of modalities across all clicks and, on the other, the usage among participants, taking into account whether the modality was used at least once.
  • ...and 1 more figures