Robust Intervention Learning from Emergency Stop Interventions
Ethan Pronovost, Khimya Khetarpal, Siddhartha Srinivasa
TL;DR
This work addresses learning from imperfect deployment-time interventions, specifically emergency-stop signals, by formulating Robust Intervention Learning (RIL) and proposing Residual Intervention Fine-Tuning (RIFT). By treating intervention feedback as an incomplete signal and regularizing updates toward a prior policy, RIFT enables robust policy improvement across various intervention strategies and prior qualities. The authors provide theoretical results showing principled improvement under certain intervention schemes and demonstrate empirical gains in multiple control tasks, especially when interventions are noisy or less informative. The approach offers a practical pathway to safer, more reliable deployment of autonomous systems by harmonizing supervisor signals with priors through residual fine-tuning.
Abstract
Human interventions are a common source of data in autonomous systems during testing. These interventions provide an important signal about where the current policy needs improvement, but are often noisy and incomplete. We define Robust Intervention Learning (RIL) as the problem of learning from intervention data while remaining robust to the quality and informativeness of the intervention signal. In the best case, interventions are precise and avoiding them is sufficient to solve the task, but in many realistic settings avoiding interventions is necessary but not sufficient for achieving good performance. We study robust intervention learning in the context of emergency stop interventions and propose Residual Intervention Fine-Tuning (RIFT), a residual fine-tuning algorithm that treats intervention feedback as an incomplete learning signal and explicitly combines it with a prior policy. By framing intervention learning as a fine-tuning problem, our approach leverages structure encoded in the prior policy to resolve ambiguity when intervention signals under-specify the task. We provide theoretical analysis characterizing conditions under which this formulation yields principled policy improvement, and identify regimes where intervention learning is expected to fail. Our experiments reveal that residual fine-tuning enables robust and consistent policy improvement across a range of intervention strategies and prior policy qualities, and highlight robust intervention learning as a promising direction for future work.