Risk Analysis and Risk Management for the Artificial Superintelligence Research and Development Process

Abstract

Artificial superintelligence (ASI) is increasingly recognized as a significant future risk. This chapter surveys established methodologies for risk analysis and risk management as they can be applied to ASI risk. For ASI risk analysis, an important technique is to model the sequences of steps that could result in ASI catastrophe. Each step can then be studied to get an overall understanding of the total risk. These models are called fault trees or event trees. To help build the models, it can be helpful to ask experts for their judgments on various parts of the model. Experts don’t always get their judgments right so it’s important to ask them carefully, using established procedures from risk analysis. For ASI risk management, there are two approaches. One is to make ASI technology safer. The other is to manage the human process of ASI research and development, in order to steer it towards safer ASI and away from dangerous ASI. Risk analysis and the related field of decision analysis can help people make better ASI risk management decisions. In particular, the analysis can help identify which options would be the most cost-effective, meaning that they would achieve the largest reduction in ASI risk for the amount of money spent on them.