Unanticipated interaction loops involving autonomous systems
Abstract
We are entering a world where unmanned vehicles will be common. They have the potential to dramatically decrease the cost of services whilst simultaneously increasing the safety record of whole industries.
Autonomous technologies will, by their very nature, shift decision making responsibility from individual humans to technology systems. The 2010 Flash Crash showed how such systems can create rare (but not inconceivably rare) and highly destructive positive feedback loops which can severely disrupt a sector.
In the case of Unmanned Air Systems (UAS), how might similar effects obstruct the development of the Commercial UAS industry? Is it conceivable that, like the high frequency trading industry at the heart of the Flash Crash, the algorithms we provide UAS to enable autonomy could decrease the risk of small incidents whilst increasing the risk of severe accidents? And if so, what is the relationship between probability and consequence of incidents?