The prospect of artificial general intelligence surpassing human cognitive abilities has emerged as the dominant concern in AI risk discussions, overshadowing biological and other existential threats in the minds of leading researchers and tech executives.
The debate hinges on a deceptively simple premise: if AI systems become superintelligent, can humans keep them aligned with human values and intentions? Experts disagree sharply on technical feasibility, the speed of recursive self-improvement, and whether current alignment strategies would hold.
Alan Turing posed the question in 1951. In a BBC radio lecture, he argued that once machines began thinking, they would "quickly outstrip human powers." He stated: "If a machine can think, it might think more intelligently than we do, and then where should we be?" Turing noted that even if humans retained the ability to cut power to machines at will, humanity would feel "greatly humbled" by the existence of entities it could no longer fully control.
Modern AI executives have echoed Turing's concern. Dario Amodei at Anthropic, Sam Altman at OpenAI, and Elon Musk at xAI have all flagged superintelligence control as a primary risk. In 2023, hundreds of AI researchers and industry figures signed a statement characterizing advanced AI as an existential threat.
But skepticism runs deep. Some biology and cybersecurity researchers argue that physical controls, human oversight, and rigorous testing make an AI-driven catastrophe less probable than popular dread suggests. Toby Ord, an Oxford researcher, estimated in his 2020 book "The Precipice" that the 100-year risk of an existential catastrophe from engineered plague is roughly 3 percent. That contrasts with much higher perceived risks attributed to AI by segments of the tech industry.
The internal disagreement is real. One Anthropic researcher recently left the field, explicitly citing concern that labs were developing superintelligent systems humans could not control. That departure signals ongoing tension within the AI development community over whether current safety approaches are adequate.
