Expecting the unexpected: Developing autonomous-system design principles for reacting to unpredicted events and conditions

Abstract

When developing autonomous systems, engineers and other stakeholders make great efforts to prepare the system for all foreseeable circumstances. However, such systems are still bound to encounter situations that were not considered at design time. For reasons like safety, cost, or ethics it is often highly desired that these new cases be handled correctly upon first encounter. In this paper, we first justify our position that there will always exist unpredicted events and conditions, driven by, e.g., new inventions in the real world, the diversity of world-wide system deployments and uses, and the possibility that multiple events that were unforeseen at design time (or overlooked, or knowingly abandoned following cost-benefit-risk calculations) will not only occur, but will occur together. We then argue that despite the unpredictability, handling such situations is indeed possible. Hence, we offer and exemplify design principles, which, when applied in advance, can improve the system's ability to deal with unpredicted situations. We conclude with a discussion of how this work and a much-needed thorough study of the unexpected can contribute toward a foundation of engineering principles for developing trustworthy next-generation autonomous systems.