CHARM: Considering Human Attributes for Reinforcement Modeling

TL;DR

This work investigates how human attributes shape feedback in RLHF and demonstrates that feedback values correlate with factors like robot experience and educational background, while delays show little relation to these attributes. The authors propose CHARM, an oracle that conditions on human characteristics in addition to task statistics to predict human feedback using an MLP, optimizing a combined loss of classification and regression. In a public-space study with 46 participants across two tasks, CHARM significantly improves prediction accuracy for both five-point and binary feedback scales compared to a task-statistics-only baseline, suggesting that incorporating human heterogeneity enhances RLHF reliability. These findings support more robust robot learning in real-world contexts and provide publicly available data and code to advance future RLHF research.

Abstract

Reinforcement Learning from Human Feedback has recently achieved significant success in various fields, and its performance is highly related to feedback quality. While much prior work acknowledged that human teachers' characteristics would affect human feedback patterns, there is little work that has closely investigated the actual effects. In this work, we designed an exploratory study investigating how human feedback patterns are associated with human characteristics. We conducted a public space study with two long horizon tasks and 46 participants. We found that feedback patterns are not only correlated with task statistics, such as rewards, but also correlated with participants' characteristics, especially robot experience and educational background. Additionally, we demonstrated that human feedback value can be more accurately predicted with human characteristics compared to only using task statistics. All human feedback and characteristics we collected, and codes for our data collection and predicting more accurate human feedback are available at https://github.com/AABL-Lab/CHARM

Figures (6)

• Figure 1: Our public study setup. The experiment was conducted in the atrium of a university building, where we recruited participants through our study setup and snack rewards. A total of 46 participants were recruited. Participants first filled out a questionnaire that covered six domains of human characteristics, and then provided human feedback to two robot tasks.
• Figure 2: Learning from human teachers vs. learning from CHARM. Human teachers provide feedback with varying delays and responses, while CHARM oracles predict this feedback by incorporating human characteristics and task statistics.
• Figure 3: Robotic tasks from mimicgen mandlekar2023mimicgen in robosuite simulatorzhu2020robosuite.
• Figure 4: Overview of distributions of feedback delay, feedback value, rewards, and human characteristic questionnaire responses for all participants. Top Row (left to right): (1a) Feedback Delay distribution; (1b) Feedback Value distribution; (1c) Binned reward distribution. Bottom Row (2a) is the mean and standard deviation of the responses across all the questions in the human characteristic questionnaire. Each bar corresponds to one question (e.g., Q1 → “Reliability”), and each x-axis label is a word that summarizes the question. Questions from the same human characteristics domain share a common color. From left to right: Trust in Robot, Robot Experience, Education Background, Teaching Experience, Teaching Style, and Personality.
• Figure 5: Domain-wise correlations between human characteristics and feedback delay, feedback accuracy, and absolute difference Each bar represents one human characteristic domain. Given that the feedback value-reward correlation is 0.563 (as the dotted line), correlations exceeding 0.1 are considered noteworthy. The figure indicates that there is no substantial linear correlation between any human characteristic domain and feedback delay. Robot experience and educational background have the most noticeable correlations with feedback accuracy and absolute difference.