Adaptive Querying for Reward Learning from Human Feedback

TL;DR

This work tackles learning to avoid negative side effects (NSEs) in robotics when reward specifications are incomplete by learning a penalty for NSEs through human feedback. It introduces Adaptive Feedback Selection (AFS), which jointly optimizes when to query (critical states) and in which feedback format to query (e.g., approvals, corrections, rankings) by maximizing information gain while accounting for human effort via a feedback preference model and a KL-divergence stopping criterion. The method maps diverse feedback into a unified NSE penalty, incorporated into the robot’s reward to produce NSE-minimizing policies, and is validated through simulation across multiple domains and through a real-world Kinova arm user study, showing improved safety, efficiency, and user trust. The results demonstrate practical benefits for human-in-the-loop learning, including reduced workload and better alignment with user preferences, with future work focusing on continuous domains and bias-aware inference to further enhance robustness and applicability.

Abstract

Learning from human feedback is a popular approach to train robots to adapt to user preferences and improve safety. Existing approaches typically consider a single querying (interaction) format when seeking human feedback and do not leverage multiple modes of user interaction with a robot. We examine how to learn a penalty function associated with unsafe behaviors using multiple forms of human feedback, by optimizing both the query state and feedback format. Our proposed adaptive feedback selection is an iterative, two-phase approach which first selects critical states for querying, and then uses information gain to select a feedback format for querying across the sampled critical states. The feedback format selection also accounts for the cost and probability of receiving feedback in a certain format. Our experiments in simulation demonstrate the sample efficiency of our approach in learning to avoid undesirable behaviors. The results of our user study with a physical robot highlight the practicality and effectiveness of adaptive feedback selection in seeking informative, user-aligned feedback that accelerate learning. Experiment videos, code and appendices are found on our website: https://tinyurl.com/AFS-learning.

Figures (16)

• Figure 1: An illustration of adaptive feedback selection. The robot arm learns to move the blue object to the white bin, without colliding with other objects in the way, by querying the human in different format across the state space.
• Figure 2: Visualization of reward learned using different feedback types. (Row 1) Black arrows indicate queries, and feedback is in speech bubbles. (Row 2)denotes high, mild, and zero penalty. Outer box is the true reward, and inner box shows the learned reward. Mismatches between the outer and inner box colors indicate incorrect learned model.
• Figure 3: Illustration of $p$ (accumulated feedback) and $q$ (generalized NSE labels) for the object delivery task. $f^*_{1:t-1}$ indicates the feedback formats selected until iteration $t-1$. indicates no NSE; indicates mild NSE; indicates severe NSE. Queried states in each iteration is highlighted in blue.
• Figure 4: Solution approach overview. The critical states $\Omega$ for querying are selected by clustering the states. A feedback format $f^*$ that maximizes information gain is selected for querying the user across $\Omega$. The NSE model is iteratively refined based on feedback. An updated policy is calculated using a penalty function $\hat{R}_N$, derived from the learned NSE model.
• Figure 5: Illustrations of evaluation domains. Red box denotes the agent and the goal location is in green.