StROL: Stabilized and Robust Online Learning from Humans

TL;DR

This work tackles robust online reward learning from humans by treating robot learning as a dynamical system with the human's true preferences $\theta^*$ as the equilibrium. It develops a Lyapunov-based stability framework to derive convergence conditions and introduces StROL, which adds a learnable correction term $\hat{g}$ to form $\tilde{g}=g+\hat{g}$, trained offline to enlarge the basins of attraction and tolerate suboptimal human inputs. The main contributions are: (i) formal convergence conditions for real-time learning from human feedback, (ii) an offline-trained correction-term mechanism that yields robust-by-design online learning dynamics, and (iii) empirical validation through simulations and a user study showing improved accuracy and reduced regret, with faster adaptation. This approach enables fast, user-specific adaptation in real-time human-robot interaction, particularly under noise, bias, and suboptimal teaching signals.

Abstract

Robots often need to learn the human's reward function online, during the current interaction. This real-time learning requires fast but approximate learning rules: when the human's behavior is noisy or suboptimal, current approximations can result in unstable robot learning. Accordingly, in this paper we seek to enhance the robustness and convergence properties of gradient descent learning rules when inferring the human's reward parameters. We model the robot's learning algorithm as a dynamical system over the human preference parameters, where the human's true (but unknown) preferences are the equilibrium point. This enables us to perform Lyapunov stability analysis to derive the conditions under which the robot's learning dynamics converge. Our proposed algorithm (StROL) uses these conditions to learn robust-by-design learning rules: given the original learning dynamics, StROL outputs a modified learning rule that now converges to the human's true parameters under a larger set of human inputs. In practice, these autonomously generated learning rules can correctly infer what the human is trying to convey, even when the human is noisy, biased, and suboptimal. Across simulations and a user study we find that StROL results in a more accurate estimate and less regret than state-of-the-art approaches for online reward learning. See videos and code here: https://github.com/VT-Collab/StROL_RAL

Figures (4)

• Figure 1: Human physically correcting a robot arm to convey their reward parameters $\theta^*$. The robot learns online, and updates its point estimate $\theta$ after each human action. (Left) When the human takes noisy or suboptimal actions, the given learning dynamics can become unstable and fail to converge to $\theta^*$. (Right) We learn how to modify these dynamics to expand the basins of attraction and increase robustness to imperfect human inputs.
• Figure 2: Example of how StROL autonomously generates robust-by-design learning rules that expand the basin of attraction. (Left) The robot does not know how it should carry a cup near a laptop. When $\theta = +1$ the human wants the robot to move straight to the goal, and when $\theta = -1$ the human wants the robot to avoid moving above the laptop. (Right) Plots of the robot's estimate $\theta$ as a function of the human's action $u_\mathcal{H}$ at the start state. With the original learning dynamics $g$ the learning is inconsistent and gradual (i.e., nearby actions can convey either ignoring or avoiding the laptop). But StROL outputs the modified learning dynamics $\tilde{g}$ to expand the basin of attraction, so that nearby actions teach the robot the same parameters.
• Figure 3: We compare StROL to state-of-the-art baselines in a multi-agent Highway environment (Top) and a collaborative Robot setting (Bottom). In Highway, the robot car takes turns interacting with $250$ simulated human cars and tries to predict whether it should change lanes. We measure the Error between the robot's learned estimate $\theta$ and the simulated human's objective $\theta^*$. In Robot, $100$ simulated humans teach a $7$ DoF Franka-Emika robot arm to reach for or avoid two stationary objects (also see Figure \ref{['fig:methods']}). We measure the Regret over the robot's learned behavior. For both environments we simulate humans with different levels of noise and bias. During offline training, e2e and StROL expected $10\%$ noise in Highway and 25% noise in Robot. The left column corresponds to this training setting. The other columns compare each method as the simulated human's noise, bias, and prior over $\theta^*$ deviates from the training data. An $*$ represents statistical significance ($p < 0.05$). A tabular version of these results is presented in our GitHub repository.
• Figure 4: Objective and subjective results from the user study in Section \ref{['sec:user']}. Participants physically interacted with a $7$-DoF robot arm (see Figure \ref{['fig:front']}) to teach it three different tasks. The robot used StROL or other online learning methods losey2022physicalbobu2020quantifying to infer the human's reward parameters in real-time. (Left) The time users spent correcting the robot and the regret across the robot's learned trajectory averaged over all three tasks. (Middle) For each individual task and participant ($3$ tasks $\times$$12$ participants) we plot their regret vs. correction time. (Right) The average user ratings from our $7$-point Likert scale survey. Error bars show SEM and an $*$ denotes statistical significance ($p < 0.05$). A tabular version is presented in our GitHub repository.