Trajectory Improvement and Reward Learning from Comparative Language Feedback

TL;DR

This work aims to incorporate comparative language feedback to iteratively improve robot trajectories and to learn reward functions that encode human preferences and leverages the learned latent space to improve trajectories and learn human preferences.

Abstract

Learning from human feedback has gained traction in fields like robotics and natural language processing in recent years. While prior works mostly rely on human feedback in the form of comparisons, language is a preferable modality that provides more informative insights into user preferences. In this work, we aim to incorporate comparative language feedback to iteratively improve robot trajectories and to learn reward functions that encode human preferences. To achieve this goal, we learn a shared latent space that integrates trajectory data and language feedback, and subsequently leverage the learned latent space to improve trajectories and learn human preferences. To the best of our knowledge, we are the first to incorporate comparative language feedback into reward learning. Our simulation experiments demonstrate the effectiveness of the learned latent space and the success of our learning algorithms. We also conduct human subject studies that show our reward learning algorithm achieves a 23.9% higher subjective score on average and is 11.3% more time-efficient compared to preference-based reward learning, underscoring the superior performance of our method. Our website is at https://liralab.usc.edu/comparative-language-feedback/

Figures (11)

• Figure 1: An image from our human subject studies where the human user wants the robot to pick up the spoon. Compared to traditional comparison preference learning, our language preference learning enables users to give more informative feedback, which helps the robot to capture human preferences more efficiently.
• Figure 2: Overview of our approach. (a) Architecture of the model that learns a shared latent space between trajectories and comparative language feedback. (b) Comparative language-based reward learning.
• Figure 3: Each dataset sample is a pair of trajectories and a language feedback.
• Figure 4: Results of experiments where we use simulated human language feedback to iteratively improve a robot trajectory (mean $\pm$ std over 100 runs). The dashed line represents average reward of optimal trajectories.
• Figure 5: Results of reward learning, averaged over 3 seeds. (a) Cross Entropy: our method converges faster. (b) True Reward of Optimal Trajectory: Our approach.