A Measure for Level of Autonomy Based on Observable System Behavior

TL;DR

The paper addresses the challenge of measuring an autonomous system's level of autonomy in real-world, observation-rich settings, where traditional metrics rely on known goals and internal states obtained during design or testing. It proposes a runtime observable measure based on two components: a human behavior lookup table and a foundational expression $L ≡ ε( ∑ H_a, ∑ O_a )$, implemented via an algorithm that compares observed action sequences to human-based equivalents using an edit-distance metric. The approach includes an algorithmic outline (Algorithm 1) and a mapping from an Observational Score (0.0–5.9) to the SAE five-level scale (and potentially to ALFUS for finer granularity), enabling cross-system comparison in the wild. While promising for runtime benchmarking and safety-aware inter-system assessment, the framework depends on sensor fidelity, robust data, and guarded handling of biased or adversarial inputs, and it points to future work in autonomic computing, cybersecurity, and broader generalization with generative AI.

Abstract

Contemporary artificial intelligence systems are pivotal in enhancing human efficiency and safety across various domains. One such domain is autonomous systems, especially in automotive and defense use cases. Artificial intelligence brings learning and enhanced decision-making to autonomy system goal-oriented behaviors and human independence. However, the lack of clear understanding of autonomy system capabilities hampers human-machine or machine-machine interaction and interdiction. This necessitates varying degrees of human involvement for safety, accountability, and explainability purposes. Yet, measuring the level autonomous capability in an autonomous system presents a challenge. Two scales of measurement exist, yet measuring autonomy presupposes a variety of elements not available in the wild. This is why existing measures for level of autonomy are operationalized only during design or test and evaluation phases. No measure for level of autonomy based on observed system behavior exists at this time. To address this, we outline a potential measure for predicting level of autonomy using observable actions. We also present an algorithm incorporating the proposed measure. The measure and algorithm have significance to researchers and practitioners interested in a method to blind compare autonomous systems at runtime. Defense-based implementations are likewise possible because counter-autonomy depends on robust identification of autonomous systems.

Figures (1)

• Figure 1: The intersection of autonomy and artificial intelligence.