Risks of the Journey to the Singularity

Abstract

Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. Unlike current AI systems, individual AGIs would be capable of learning to operate in a wide variety of domains, including ones they had not been specifically designed for. It has been proposed that AGIs might eventually pose a significant risk to humanity, for they could accumulate significant amounts of power and influence in society while being indifferent to what humans valued. The accumulation of power might either happen gradually over time, or it might happen very rapidly (a so-called “hard takeoff”). Gradual accumulation would happen through normal economic mechanisms, as AGIs came to carry out an increasing share of economic tasks. A hard takeoff could be possible if AGIs required significantly less hardware to run than was available, or if they could redesign themselves to run at ever faster speeds, or if they could repeatedly redesign themselves into more intelligent versions of themselves.