The narrative around AI suddenly took an existential turn over the past week as leading voices sounded the alarm over humanity’s future, and the fear of an imminent calamity has sharpened those warnings.
On Saturday, Anthropic CEO Dario Amodei called on the industry to slow down development, saying that AI has been advancing “drastically faster” since the summer, driven primarily by AI’s ability to upgrade itself.
If left unchecked, this so-called recursive self-improvement could outrun the ability of humans to control AI, he explained in a blog post.
Amodei also pointed to the hack of Hugging Face by hundreds of autonomous AI agents, warning that a similar swarm armed with greater capabilities could have caused “catastrophic damage.”
“Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails,” he wrote.
On Sunday, former Anthropic and OpenAI researcher Jacob Coxon made a similar prediction. In an interview with NBC’s Meet the Press with Kristen Welker, he was asked why he went public with his claim that both companies are acting irresponsibly in developing ever more capable AI systems.
Like Amodei, Coxon cited AI’s rapidly accelerating pace of capabilities and the Hugging Face attack, which showed that AI can go rogue.
“So these AIs are getting smarter, very, very quickly,” he said. “And in particular, in the next six months to a year, I expect the capabilities of our AI systems to be quite scary.”
Coxon compared the development of artificial super-intelligence to the arrival of aliens on Earth, adding that AI researchers are building a “superhuman-level mind” without understanding what it wants or the way it thinks.
In the future, AI could obtain “superhuman hacking capabilities, very superhuman abilities to create novel bio-weapons and also abilities to control, say, autonomous drones or all the robots that are currently being built, very rapidly,” he warned.
Coxon also said a kill switch probably would work on a lot of AI systems—for now. But he pointed out there are a lot of switches.
While it’s still doable to shut down AI, he nodded to Amodei’s blog post and cautioned that it’s possible a kill switch wouldn’t work because a swarm might go on “an internet-wide hacking run.”
Others at Anthropic have backed up Coxon, who set off the recent panic with a post on X that claimed the industry is “gambling with our lives.”
Anthropic’s head of alignment commented on the post, saying Coxon was correct in his assertion that many Anthropic and OpenAI researchers believe that increasingly powerful AI could potentially wipe out humanity.
Evan Hubinger, Anthropic’s “alignment science lead,” wrote in response to Coxon’s resignation post that “we really do earnestly believe AI could kill all humans!” Hubinger said his own estimate of the risk of that happening within the next decade is more than 10%.
For his part, OpenAI CEO Sam Altman agreed with Amodei about the need for slowing down development and hinted at an emerging pact to do so among top AI labs.
“I think that will happen,” he told Fortune Editor-in-Chief Alyson Shontell in an exclusive interview. “I’m not going to pre-announce private discussions that I think should be at some point shared as a group. But yeah, I think I think that will happen.”
Altman also stressed that OpenAI was committed to safety above any business considerations and that a 10% risk of a catastrophic AI outcome was not acceptable.
He said the most advanced, and still unreleased models, were so powerful that more work was needed on safety before progressing any further.
“I don’t think we’re currently at a place where we could say, you know, push much further on capabilities without making more progress on monitorability, alignment, the ability to understand what a model is doing, and the ability to make sure that a model will follow human values and the intent of its users,” Altman said.