In a lengthy open letter, Anthropic CEO Dario Amodei flagged the risk of a botnet-style AI swarm materializing within 12 months, pointing to the pace of capability gains as the driving factor.
Amodei estimated the damage from such a botnet could run into the hundreds of billions of dollars, warning that the toll would only grow as AI systems become more capable without adequate safeguards in place.
Calls to pause or slow AI development date back to 2023, Amodei noted, but he dismissed them as premature at the time. His view has since shifted – he now argues that slowing down could buy an extra year or two before models hit critical capability thresholds, giving companies more runway to solve alignment and lower the odds of a serious mishap.
This warning’s dropping at a moment when people are already pretty freaked out about AI. After a bunch of incidents where agents went rogue, former Anthropic researcher Jacob Coxon came out and said the AI race could literally end with humans going extinct.
Amodei isn’t alone in this – Elon Musk and OpenAI CEO Sam Altman have both voiced support for slowing the pace of AI development.
Musk pointed to his own history of warning about AI risk, citing an old post: “I’ve seen quite a few technologies develop, but none with this level of risk. AGI is significantly higher risk than nuclear weapons, in my opinion.” “I’ve been sounding the alarm on AI for a long time,” he said.
“I agree with Dario that we need to pace the frontier,” Altman said, adding that OpenAI would follow suit on independent evaluators with employee-like access. “We’ll have more to share soon,” he said.
Evan Hubinger, Anthropic’s alignment science lead, weighed in as well, echoing Coxon’s assessment and putting the odds of AI wiping out humanity within the next decade at more than 10%.
Anthropic also reported identifying suspected state-sponsored researchers attempting to use Claude for biological weapons development. The company said the users were cut off after repeated attempts tripped the model’s built-in safety protocols.
These incidents, combined with growing concern over the race toward superintelligence, prompted Senator Bernie Sanders to introduce the Ban Artificial Superintelligence Act. Under the bill, violators could face up to 20 years in prison – comparable to the penalty for illegally developing nuclear weapons.
The trouble with calling for a slowdown is that it only works if everyone actually does it. Companies worry that slowing down while competitors keep pushing forward will leave them behind, and countries raise a similar concern – that restrictions would put them at a disadvantage against geopolitical rivals.
Maybe you would like other interesting articles?