A Quantitative Autonomy Quantification Framework for Fully Autonomous Robotic Systems
Nasser Gyagenda, Hubert Roth
TL;DR
The paper tackles the challenge of robustly quantifying autonomous capability by introducing a task-based framework with three metrics: requisite capability set, reliability, and responsiveness. It couples these with a two-part autonomy measure—Level of Autonomy (LoA, ordinal) and Degree of Autonomy (DoA, ratio)—and an online integrity monitor to ensure safe operation. The approach is demonstrated on autonomous driving and DARPA Subterranean Challenge rules, showing how LoA/DoA discriminate performance beyond simple level classifications and how integrity monitoring provides regulatory and safety leverage. This framework supports design guidance, standardization, and trust-building for fully autonomous systems, with potential extension to multi-agent settings and online regulation.
Abstract
Although autonomous functioning facilitates deployment of robotic systems in domains that admit limited human oversight on our planet and beyond, finding correspondence between task requirements and autonomous capability is still an open challenge. Consequently, a number of methods for quantifying autonomy have been proposed over the last three decades, but to our knowledge all these have no discernment of sub-mode features of variation of autonomy and some are based on metrics that violet the Goodhart's law. This paper focuses on the full autonomous mode and proposes a quantitative autonomy assessment framework based on task requirements. The framework starts by establishing robot task characteristics from which three autonomy metrics, namely requisite capability set, reliability and responsiveness are derived. These characteristics were founded on the realization that robots ultimately replace human skilled workers, from which a relationship between human job and robot task characteristics was established. Additionally, mathematical functions mapping metrics to autonomy as a two-part measure, namely of level and degree of autonomy are also presented. The distinction between level and degree of autonomy stemmed from the acknowledgment that autonomy is not just a question of existence, but also one of performance of requisite capability. The framework has been demonstrated on two case studies, namely autonomous vehicle at an on-road dynamic driving task and the DARPA subterranean challenge rules analysis. The framework provides not only a tool for quantifying autonomy, but also a regulatory interface and common language for autonomous systems developers and users. Its greatest feature is the ability to monitor system integrity when implemented online.