ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning

TL;DR

ThriftyDAgger presents a budget-aware, robot-gated interactive imitation learning framework that gates human interventions using jointly learned novelty and risk signals. By automatically tuning thresholds from a user-specified intervention budget, it reduces supervisor burden while maintaining or improving task performance across simulation, fleet-control, and physical visuomotor tasks. The approach demonstrates strong autonomous performance during execution, achieves high intervention efficiency, and lowers operator workload in user studies, marking a practical advance for scalable interactive learning in robotics.

Abstract

Effective robot learning often requires online human feedback and interventions that can cost significant human time, giving rise to the central challenge in interactive imitation learning: is it possible to control the timing and length of interventions to both facilitate learning and limit burden on the human supervisor? This paper presents ThriftyDAgger, an algorithm for actively querying a human supervisor given a desired budget of human interventions. ThriftyDAgger uses a learned switching policy to solicit interventions only at states that are sufficiently (1) novel, where the robot policy has no reference behavior to imitate, or (2) risky, where the robot has low confidence in task completion. To detect the latter, we introduce a novel metric for estimating risk under the current robot policy. Experiments in simulation and on a physical cable routing experiment suggest that ThriftyDAgger's intervention criteria balances task performance and supervisor burden more effectively than prior algorithms. ThriftyDAgger can also be applied at execution time, where it achieves a 100% success rate on both the simulation and physical tasks. A user study (N=10) in which users control a three-robot fleet while also performing a concentration task suggests that ThriftyDAgger increases human and robot performance by 58% and 80% respectively compared to the next best algorithm while reducing supervisor burden.

Figures (4)

• Figure 1: Left: An example start and goal state for the block stacking environment in Robosuite. The goal is to place the red block on top of the green one. Initial poses of both blocks are randomized. Right: The da Vinci Research Kit Master Tool Manipulator (MTM) 7DOF interface used to provide human interventions in the physical experiments. The human expert views the workspace through the viewer (top) and teleoperates the robot by moving the right joystick (middle) in free space while pressing the rightmost pedal (bottom).
• Figure 2: User Study Survey Results: We illustrate the user study interface for the human-gated and robot-gated algorithms (left) and users' survey responses regarding their mental load and frustration (right) for each algorithm. Results suggest that users report similar levels of mental load and frustration for ThriftyDAgger and LazyDAgger, but significantly higher levels of both metrics for HG-DAgger and SafeDAgger. We hypothesize that the sparing and sustained interventions solicited by ThriftyDAgger and LazyDAgger lead to greater user satisfaction and comfort compared to algorithms which force the user to constantly monitor the system (HG-DAgger) or frequently context switch between teleoperation and the distractor task.
• Figure 3: ThriftyDAgger: Given a desired context switching rate $\alpha_{h}$, ThriftyDAgger transfers control to a human supervisor if the current state $s_t$ is (1) sufficiently novel or (2) sufficiently risky, indicating that the probability of task success is low under robot policy $\pi_r$. Intuitively, one should not only distrust $\pi_r$ in states significantly out of the distribution of previously-encountered states, but should also cede control to a human supervisor in more familiar states where the robot predicts that it is unlikely to successfully complete the task.
• Figure 4: Experimental Domains: We visualize the peg insertion simulation domain (top row) and the physical cable routing domain with the physical robot (bottom row). We visualize sample start and goal states, in addition to states which ThriftyDAgger categorizes as novel, risky, and neither. ThriftyDAgger marks states as novel if they are far from behavior that the supervisor would produce, and risky if it is stuck in a bottleneck, e.g. if the ring is wedged against the side of the cylinder (top) or the cable is near all four obstacles (bottom).