Robot Learning as an Empirical Science: Best Practices for Policy Evaluation
Hadas Kress-Gazit, Kunimatsu Hashimoto, Naveen Kuppuswamy, Paarth Shah, Phoebe Horgan, Gordon Richardson, Siyuan Feng, Benjamin Burchfiel
TL;DR
This paper argues that robot-learning evaluation must move beyond sole reliance on success rate to achieve reproducible, informative insights. It proposes a comprehensive best-practices framework encompassing explicit success criteria, controlled initial conditions, and interleaved AB testing; it introduces semantic metrics (rubrics and Signal Temporal Logic) and performance measures (STL robustness, SPARC smoothness) to capture task nuance. The authors stress rigorous reporting of experimental parameters, statistical analyses (including Bayesian methods), and detailed failure-mode descriptions, illustrated with physical manipulation tasks and an exemplar evaluation report. The guidelines aim to improve rigor, comparability, and understanding of failure modes, with applicability to both physical experiments and simulations and a push toward open evaluation data.
Abstract
The robot learning community has made great strides in recent years, proposing new architectures and showcasing impressive new capabilities; however, the dominant metric used in the literature, especially for physical experiments, is "success rate", i.e. the percentage of runs that were successful. Furthermore, it is common for papers to report this number with little to no information regarding the number of runs, the initial conditions, and the success criteria, little to no narrative description of the behaviors and failures observed, and little to no statistical analysis of the findings. In this paper we argue that to move the field forward, researchers should provide a nuanced evaluation of their methods, especially when evaluating and comparing learned policies on physical robots. To do so, we propose best practices for future evaluations: explicitly reporting the experimental conditions, evaluating several metrics designed to complement success rate, conducting statistical analysis, and adding a qualitative description of failures modes. We illustrate these through an evaluation on physical robots of several learned policies for manipulation tasks.