Expecting the Unexpected: Developing Autonomous-System Design Principles for Reacting to Unpredicted Events and Conditions
Assaf Marron, Lior Limonad, Sarah Pollack, David Harel
TL;DR
The paper tackles the problem of unpredicted events in autonomous systems by arguing that first-encounter handling is achievable through an Autonomics-inspired engineering foundation. It proposes a structured set of design principles across reactive/proactive behaviors, knowledge management, and social-ecosystem considerations, augmented by proactive domain exploration. Key contributions include concrete reaction rules, look-ahead and probing mechanisms, knowledge-sharing practices, and negotiation and mimicry strategies to operate safely in unknown conditions. The work aims to establish a rigorous, theory-grounded framework with ontologies and digital-twin notions to mature trustworthy next-generation autonomous systems, enabling robust performance as real-world environments evolve. Its practical impact lies in guiding developers toward architectures and processes that anticipate the unknown, improving safety, reliability, and adaptability of autonomous technologies.
Abstract
When developing autonomous systems, engineers and other stakeholders make great effort to prepare the system for all foreseeable events and conditions. However, these systems are still bound to encounter events and conditions that were not considered at design time. For reasons like safety, cost, or ethics, it is often highly desired that these new situations be handled correctly upon first encounter. In this paper we first justify our position that there will always exist unpredicted events and conditions, driven among others by: new inventions in the real world; the diversity of world-wide system deployments and uses; and, the non-negligible probability that multiple seemingly unlikely events, which may be neglected at design time, will not only occur, but occur together. We then argue that despite this unpredictability property, handling these events and conditions is indeed possible. Hence, we offer and exemplify design principles that when applied in advance, can enable systems to deal, in the future, with unpredicted circumstances. We conclude with a discussion of how this work and a broader theoretical study of the unexpected can contribute toward a foundation of engineering principles for developing trustworthy next-generation autonomous systems.