To Err is Robotic: Rapid Value-Based Trial-and-Error during Deployment

TL;DR

The paper tackles rapid adaptation of robot policies to novel scenarios by embedding explicit trial-and-error behavior into an expert-trained base policy. It introduces Bellman-Guided Retrials, which attach a value function learned from demonstrations to monitor progress and trigger recoveries and retrials when suboptimal progress is detected via the Bellman target $y = r + \gamma V_\phi(s_t)$. A non-parametric skewing mechanism biases future samples away from past failure states, enabling diverse strategy exploration without requiring reward signals or data collection beyond the original demonstrations. Across simulated grasping tasks and real-robot experiments, the approach yields significant performance gains (over $20\%$ in simulation and over $50\%$ real-world) and demonstrates robust assistance across short-horizon and long-horizon tasks. The framework provides a practical, data-efficient pathway to retrofit existing policies with explicit deployment-time trial-and-error capabilities, while highlighting limitations related to recovery policies and environment dynamics.

Abstract

When faced with a novel scenario, it can be hard to succeed on the first attempt. In these challenging situations, it is important to know how to retry quickly and meaningfully. Retrying behavior can emerge naturally in robots trained on diverse data, but such robot policies will typically only exhibit undirected retrying behavior and may not terminate a suboptimal approach before an unrecoverable mistake. We can improve these robot policies by instilling an explicit ability to try, evaluate, and retry a diverse range of strategies. We introduce Bellman-Guided Retrials, an algorithm that works on top of a base robot policy by monitoring the robot's progress, detecting when a change of plan is needed, and adapting the executed strategy until the robot succeeds. We start with a base policy trained on expert demonstrations of a variety of scenarios. Then, using the same expert demonstrations, we train a value function to estimate task completion. During test time, we use the value function to compare our expected rate of progress to our achieved rate of progress. If our current strategy fails to make progress at a reasonable rate, we recover the robot and sample a new strategy from the base policy while skewing it away from behaviors that have recently failed. We evaluate our method on simulated and real-world environments that contain a diverse suite of scenarios. We find that Bellman-Guided Retrials increases the average absolute success rates of base policies by more than 20% in simulation and 50% in real-world experiments, demonstrating a promising framework for instilling existing trained policies with explicit trial and error capabilities. For evaluation videos and other documentation, go to https://sites.google.com/view/to-err-robotic/home

Figures (10)

• Figure 1: The Bellman-Guided Retrials Method. (A) Using expert demonstrations, we train a policy and value function. While solving a novel situation, we evaluate progress by looking at the behavior of the trained value function (B). If we detect suboptimality, we recover the robot and perform (C), which modifies the sampling of the pretrained policy to avoid past mistake states.
• Figure 2: Experiment Environments. We consider four sets of experiment domains spanning simulated and real tasks of varying difficulty.
• Figure 3: Visualizing Suboptimality Detection. The change in value function (Eq \ref{['progress']}) drops below zero immediately as the object slips out of the robot's grasp.
• Figure 4: Comparing Bellman-Guided Retrials To Relevant Baselines. In both simulation and real robots, our method boosts the performance of the base policy and outperforms an interval recovery baseline. Note that the DoorOpening results are limited because we lost the environment prematurely (Appendix \ref{['addl_exp_robot']}).
• Figure 5: Details on training and using the value function. We train $V_\phi$ to output a categorical distribution (B) by regressing the logit outputs to a softened one-hot vector (A). During test-time, we compute $V(s_t) - V(s_{t-k})$ by convolving the distributions and computing an upper bound on the difference (C). We show an example of acceptable progress (green, C) and not acceptable progress (red, C).