Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What you reward is what you learn: Comparing rewards for online speech policy optimization in public HRI

Sichao Song, Yuki Okafuji, Kaito Ariu, Amy Koike

TL;DR

This paper compares three complementary binary rewards--Ru, Rc, and Rt--and shows that each induces distinct arm distributions and interaction behaviors, and distill ready-to-use design lessons for deploying online optimization of speech policies in real public HRI settings.

Abstract

Designing policies that are both efficient and acceptable for conversational service robots in open and diverse environments is non-trivial. Unlike fixed, hand-tuned parameters, online learning can adapt to non-stationary conditions. In this paper, we study how to adapt a social robot's speech policy in the wild. During a 12-day in-situ deployment with over 1,400 public encounters, we cast online policy optimization as a multi-armed bandit problem and use Thompson sampling to select among six actions defined by speech rate (slow/normal/fast) and verbosity (concise/detailed). We compare three complementary binary rewards--Ru (user rating), Rc (conversation closure), and Rt (>=2 turns)--and show that each induces distinct arm distributions and interaction behaviors. We complement the online results with offline evaluations that analyze contextual factors (e.g., crowd level, group size) using video-annotated data. Taken together, we distill ready-to-use design lessons for deploying online optimization of speech policies in real public HRI settings.

What you reward is what you learn: Comparing rewards for online speech policy optimization in public HRI

TL;DR

This paper compares three complementary binary rewards--Ru, Rc, and Rt--and shows that each induces distinct arm distributions and interaction behaviors, and distill ready-to-use design lessons for deploying online optimization of speech policies in real public HRI settings.

Abstract

Designing policies that are both efficient and acceptable for conversational service robots in open and diverse environments is non-trivial. Unlike fixed, hand-tuned parameters, online learning can adapt to non-stationary conditions. In this paper, we study how to adapt a social robot's speech policy in the wild. During a 12-day in-situ deployment with over 1,400 public encounters, we cast online policy optimization as a multi-armed bandit problem and use Thompson sampling to select among six actions defined by speech rate (slow/normal/fast) and verbosity (concise/detailed). We compare three complementary binary rewards--Ru (user rating), Rc (conversation closure), and Rt (>=2 turns)--and show that each induces distinct arm distributions and interaction behaviors. We complement the online results with offline evaluations that analyze contextual factors (e.g., crowd level, group size) using video-annotated data. Taken together, we distill ready-to-use design lessons for deploying online optimization of speech policies in real public HRI settings.
Paper Structure (31 sections, 2 equations, 4 figures, 4 tables, 1 algorithm)

This paper contains 31 sections, 2 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Thompson Sampling (TS)
  4. Bernoulli rewards with Beta priors.
  5. Experiment Design
  6. Field and Tasks
  7. Reward Design
  8. Data Collection
  9. Service Robot System
  10. Thompson Sampling Integration
  11. Evaluation and Analysis
  12. Bandit Performance
  13. Video Annotation
  14. Results
  15. Bandit Performance
  16. ...and 16 more sections

Figures (4)

  • Figure 1: Sota robot designed for route guidance and field setting in the shopping mall. It features an integrated display. The display explicitly communicates the robot’s ability to provide directional assistance, ensuring users are aware of its functionality.
  • Figure 2: This diagram shows a robot's speech policy change using Thompson sampling (TS). After the interaction ends, three types of reward conditions are given. In the $R_u$ (User rating) condition, the user who interacted with the robot evaluates the robot using a questionnaire, while in the $R_c$ (Conversation closure) and $R_t$ ($\geq$ 2 turns) conditions, a remote third party evaluates the objective metrics of interaction. The obtained evaluation as a reward is sent to the robot system in real time, and the robot's next behavior is determined using TS and immediately reflected.
  • Figure 3:
  • Figure 4: Cumulative reward curves (Thompson Sampling vs. Uniform Sampling).