Training People to Reward Robots

TL;DR

The paper tackles the challenge of training novices to teach robots in RLfD by introducing a Machine Teaching (MT) framework that guides novice teachers to provide high-quality reward demonstrations with minimal data. It combines LSPI-based RLfD with a bilevel MT objective, and operationalizes this through a scaffolding training interface that visualizes ideal rewards and progressively increases task complexity. The authors demonstrate that MT-guided training significantly improves demonstration quality on trained tasks and transfers to unseen tasks, with pronounced gains for long-horizon skills, and show tradeoffs with supervised schemes on shorter horizons. This approach offers a practical path to upskill workers for robot learning systems, reducing reliance on expert demonstrators and enabling scalable, robust RLfD deployment.

Abstract

Learning from demonstration (LfD) is a technique that allows expert teachers to teach task-oriented skills to robotic systems. However, the most effective way of guiding novice teachers to approach expert-level demonstrations quantitatively for specific teaching tasks remains an open question. To this end, this paper investigates the use of machine teaching (MT) to guide novice teachers to improve their teaching skills based on reinforcement learning from demonstration (RLfD). The paper reports an experiment in which novices receive MT-derived guidance to train their ability to teach a given motor skill with only 8 demonstrations and generalise this to previously unseen ones. Results indicate that the MT-guidance not only enhances robot learning performance by 89% on the training skill but also causes a 70% improvement in robot learning performance on skills not seen by subjects during training. These findings highlight the effectiveness of MT-guidance in upskilling human teaching behaviours, ultimately improving demonstration quality in RLfD.

Figures (6)

• Figure 1: Overview of training workflow. The novice user provides low-quality demonstrations, causing the robot to fail at the training task after (light orange box). Using the known optimal solution to the training task (green box), can generate guidance to be used in the training framework. Once the user has been trained, they can provide rewards more effectively, leading to improved learning outcomes for the robot both on the training task (orange box) and for new tasks not seen in training (blue box).
• Figure 2: Visual training interface setup for example training task S\ref{['skill:1']} (see §\ref{['s:evaluation']}). Trainee teachers are required to click and drag the red slider to provide reward demonstrations (orange arrow) for the key-frame state-action pair presented (black dot, red arrow). Feedback is provided in terms of the ideal reward $\bar{\jmath}$ (blue bar). To aid users in calibrating the scale of rewards, the robot's workspace is demarked with a grid (black dashed lines) and a representation of the maximum admissible actions (red circle).
• Figure 3: Scaffolding training (P\ref{['phase:3']}–P\ref{['phase:7']}) for S\ref{['skill:1']} (see §\ref{['s:evaluation']}). The ideal reward for the illustrated state (black dots)-action (red arrows) pairs is shown in blue on the slider at the bottom. Example state-action-reward pairs are shown in sequence, with the bold one representing the 1st demonstration, while the faded ones represent future demonstrations.
• Figure 4: Comparison of subjects' performance in teaching S\ref{['skill:1']}: Top view of a trajectory taught by a representative subject from the target group in P\ref{['phase:1']} (dotted orange line) versus P\ref{['phase:9']} (dashed orange line), where the gripper is reaching to place an object at the target in the bowl.$\mathnormal{E_{\acrshort{ADE}}}$ and$\mathnormal{E_{ARMSE}}$ in P\ref{['phase:1']} and P\ref{['phase:9']}, for the target (black box) and control (white box) groups. The red lines are medians. Black circles indicate outliers (defined as those lying outside the $0$-$90$ percentile range).
• Figure 5: Comparison of subjects' performance in teaching S\ref{['skill:2']}: Top view of a trajectory taught by a representative subject from the target group in P\ref{['phase:2']} (dotted orange line) versus P\ref{['phase:8']} (dashed orange line), where the gripper is reaching to place an object onto the conveyor belt (red line).$\mathnormal{E_{\acrshort{ADE}}}$, and$\mathnormal{E_{ARMSE}}$ in P\ref{['phase:2']} and P\ref{['phase:8']}, for the target (black box) and control (white box) groups. Black circles indicate outliers (defined as those lying outside the $0$-$90$ percentile range).